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EXECUTIVE SUMMARY 


Virginia Department of Education (VDOE) was awarded a four-year Striving Readers grant from 
the U.S. Department of Education (USED) to start in school year (SY) 2009-2010. The Striving 
Readers program funded studies on the impact of supplemental reading interventions on 
adolescents whose reading skills were below grade level. 


The VDOE project, called Virginia Striving Readers Intervention Initiative (VSRID), focused on 
the implementation and impact of Passport Reading Journeys (PRJ) for seventh and eighth grade 
students who were in need of further reading instruction. PRJ, published by Cambium Learning 
Group, is a supplemental reading intervention offered to students from grades six to nine. For 
each grade level, the topic areas remain the same but are explored from different perspectives 
and with grade-appropriate content. The intervention is organized into 15 Expedition lesson 
series taught daily in 50-minute periods during one school year. Each Expedition comprises two 
two-day sequences that include a day of teacher-directed whole group instruction followed by a 
day of whole group review and small group instruction. This sequence is then followed by a 
fifth day of student individualized computer-based practice. After the fifth day, the two two-day 
sequence and a day of individualized practice are repeated to complete the ten lessons that form 
each Expedition. The materials for the intervention provided by PRJ include teacher 
guidebooks, student workbooks, DVDs, and a library of fiction and non-fiction books and 
magazines that are age-appropriate and intended to engage the adolescent reader. 


VSRII served students at nine middle schools located in three school divisions in the east, 
central, and west part of the state. Eligibility for participation in the study included students who 
scored at least two years below grade level on the Gates-MacGinitie (GMRT) 4" Edition when in 
the sixth and seventh grades. Also eligible were students in grades 6 and 7 who did not attain the 
proficient performance level on the Virginia Standards of Learning (SOL) English/ Reading 
assessment, regardless of their GMRT score. A total of 913 students were eligible for the study. 
Eligible students were randomly assigned to treatment and control groups. The treatment group 
was instructed using PRJ, while the control group received no supplemental reading instruction. 
RMC Research Corporation (RMC) was responsible for the VSRII implementation evaluation 
and impact study. Data for the implementation evaluation were collected through interviews, 

site visits, and review of documents. The impact study focused on results from the GMRT 
(comprehension, vocabulary, and total reading) and the SOL English/Reading assessment for 
control and treatment students. 


On April 12, 2011, Striving Readers grantees were informed that Congress had not appropriated 
funding for the 2011 Fiscal Year and the study was halted. With the first year (September 2009 
to June 2010) dedicated to planning, PRJ implementation was limited to one year (September 
2010 to June 2011). This report reflects findings from this initial year of implementation and the 
preliminary impact of the intervention on treatment students compared to control students. A 
summary of key findings are presented next. 


Virginia Striving Readers Intervention Initiative, Year One Implementation iv 


Year One Implementation Study 


Findings from the first implementation year suggest that the interventionists received the 
planned professional development and supports, with the exception of one school, where 
the interventionist left the school before the end of the year. 


The delivery of classroom instruction was similar across interventionists. However, the 
interventionists varied on the amount of material covered (measured as Expeditions 
completed). 


Three factors found to influence the implementation across all participating schools: (1) 
the planning year facilitated the implementation of the project by allowing the 
implementers to familiarize themselves with the intervention and the study, and opening 
the lines of communication across all participants; (2) the professional development and 
supports was a second “general” positive contributor that enhanced teachers’ knowledge 
of the program and helped them move from learning into implementing within a short 
period of time; (3) alternatively, the elimination of the Striving Readers grant before the 
end of the school year created a challenge to the implementation as teachers and key 
players started looking for jobs. 


Factors that appear to be specific to one or a small group of schools included: (1) actual 
time of instruction, including instruction periods below the required 50 minutes; (2) days 
dedicated to instruction, defined as total school days minus days of cancelled instruction; 
(3) technology glitches that influenced actual time of instruction; and (3) classroom 
management or teachers’ ability to engage students. Data from the first implementation 
year suggest that these “specific factors” had moderator roles, either reducing or 
strengthening the impact of the general factors on the fidelity of implementation. 


Year One Impact Study 
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A total of 913 students were eligible for the study. Of these, 481 students were in grade 7 
and 432 in grade 8. The eligible students were then randomly assigned to treatment (457 
students) and control (456 students) groups. 


Students in the treatment and control groups performed equally well on both the GMRT 
and the SOL at the end of the first implementation year. After adjusting for the student 
level covariates that were retained in the final analysis model, the difference between the 
two groups on all test scores was less than two scale score points. Similarly, the effect 
sizes were virtually zero. 


When the analysis was disaggregated by grade, the pattern of non-significant results was 
repeated, except for the GMRT Comprehension subtest. For this subtest, grade 7 students 
in the treatment group did significantly better (p < 0.05) than control students, but the 
effect size (0.21) was relatively small. 


INTRODUCTION AND BACKGROUND 


By the end of school year (SY) 2008-2009, Virginia Department of Education (VDOE) applied 
for and was awarded a four-year Striving Readers grant to implement the Virginia Striving 
Readers Intervention Initiative (VSRID. VSRII proposed to implement a supplemental reading 
intervention with students in seventh and eighth grades at nine public schools in three school 
divisions in Virginia. The school division representatives chose to implement Passport Reading 
Journeys (PRJ), an intervention that was already in use in many Virginian schools. PRJ had 
been studied previously in other school districts using quasi-experimental designs, but had not 
been tested with an experimental study. A total of 913 students were eligible to participate. This 
report presents provisional findings from the first implementation year of VSRII (SY 2010-2011) 
and its preliminary impact on participating students. The report is organized in four parts: 


e Introduction and Background places the study within a conceptual and geographical 
framework. It introduces the reader to the Striving Readers program and VDOE’s 
previous participation in reading initiatives, and briefly describes the schools that 
participated in the study. 

e Part I offers an overview of the intervention, as proposed by the developers, and 
describes the logic model that informed the VSRII. 

e Part II focuses on the implementation study. It includes a description of the study design 
and methods of data collection and analysis, and discusses findings. 

e Part III describes the design for the impact study and presents findings. 


The report is supplemented by two Appendices. Appendix A includes the final model results for 
the impact analysis. Appendix B includes copies of the forms used for data collection for the 
implementation study. It is important to emphasize that findings are preliminary since the study, 
originally planned for three years, was interrupted at the end of its first implementation year. 


Background 


Striving Readers was a U.S. Department of Education (USED) program that reflected a joint 
effort from the Office of Elementary and Second Education (OESE) and the Institute of 
Education Sciences (IES). The program had a dual purpose: (a) improve the reading skills of 
middle and high-school students who were reading below grade level, and (b) build a scientific 
base to identify effective strategies that improve adolescent literacy skills. Striving Readers was 
geared to Title I eligible schools that have significant percentages of students reading below 
grade level and/or schools that were not meeting or at-risk of not meeting adequate yearly 
progress (AYP) requirements under the No Child Left Behind Act (NCLB). The program 
included three key components: (a) supplemental literacy interventions targeted to students who 
are reading “significantly below grade level;” (b) cross-disciplinary strategies for improving 
adolescent literacy, including professional development and research-based reading and 
comprehension strategies; and (c) an evaluation component that uses an experimental design 
(USED, 2008). 
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The Commonwealth of Virginia has a long tradition in providing support to reading initiatives. 
In 1997, the state leveraged resources to implement the Early Intervention Reading Initiative, 
based on the work of Reid Lyon, Connie Juel, and Marilyn Adams (Wright, 2007). To support 
the initiative, researchers from the Curry School of Education, at the University of Virginia, 
developed the Phonological Awareness Literacy Screening (PALS). PALS is a research-based 
assessment of literacy fundamentals in children from preschool to grade three. The assessment is 
now used on almost all Virginian schools and early literacy programs, as well as schools around 
the country.’ 


Continuing with this focus on early intervention for struggling readers, in 2002, Virginia’s 
governor launched the Partnership for Achieving Successful Schools (PASS). PASS represented 
a statewide partnership of government officials, state and local school educators, and business 
and community leaders who shared a common concern with boosting student achievement in 
more than 100 academically struggling schools. Virginia was also one of the earliest recipients 
of a USED Reading First grant, in 2003. The Reading First Initiative involved about 90 
elementary schools across the state. In 2007, Virginia was one of three states to be awarded the 
USED Reading First Targeted Assistance Grant (TAG) for demonstrating increased reading 
achievement over two consecutive years. 


Adolescent literacy was the theme of the 2007 Virginia Board of Education summit, “Closing the 
Achievement Gap: A Focus on Adolescent Literacy.” The following year, Virginia’s Governor 
assumed the chair of the Southern Regional Education Board (SREB) Committee to Improve 
Reading and Writing in Middle and High Schools. The Committee proposed strategies geared 
toward improving reading skills for adolescent struggling readers at public schools. The Striving 
Readers program aligned with this continuum of initiatives for improving literacy for students 
from early childhood to grade 12. 


Process 


VSRII built on VDOE’s experience with the previous work and the lines of communication 
between the state and the school divisions that had been strengthened through the Reading First 
project. Representatives from VDOE, the school divisions, and RMC organized a planning team 
to prepare for the response to the Striving Readers’ Request for Proposal (RFP) in 2008. The 
team had three main goals: select the intervention to be implemented, organize the study, and 
write the proposal. Each team member explored a number of adolescent supplemental reading 
interventions that were based upon reading research and had been studied with the use of 
rigorous evaluation methods. WDOE and RMC staff helped with preparing a list of reading 
interventions that qualified under the Striving Readers program requirements, and answering 
research-related questions about those programs. Yet, the final selection was made by the school 
divisions, in conversation with the schools. One of the participating school divisions had 
previous experience with PRJ and helped the other two school divisions come to a consensus. 


PRJ is a supplemental reading intervention for students in grades 6 through 9. The intervention 
is planned for daily, 50-minute period lessons throughout the school year. PRJ Beginnings is 
offered to students in grade 6, PRJ / is for students in grade 7, while students in grade 8 are 


' Information on PALS can be found at https://www.palsmarketplace.com/about/ 
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taught with PRJ IJ and those in grade 9 are taught with PRJ I/I. The intervention provides a 
standard protocol, easy-to-follow lesson plans, an assessment system, and supporting materials 
for teachers and students. The lessons offer a mix of teacher-directed whole group, small group 
instruction and student individualized practice organized in Expeditions that focus on themes of 
interest for adolescent readers. For example, Expedition 3 of PRJ J, “What’s Out There?” is 
organized around the theme of space exploration and search for life outside the solar system. 
Expedition 11, “Things in Motion,” includes readings on acceleration using a number of topics 
popular among adolescents, such as bikes, motorcycles, skateboarding, baseball, and the 
breaking of the sound barrier. The intervention presents itself as open to a diverse student 
population, including students with disabilities and English language learners, and does not 
propose a minimum reading level for participants. 


Addressing the Striving Readers requirements, PRJ is based on findings from reading research 
and is being implemented in many school districts nationwide, including Virginia. It has been 
studied through the use of quasi-experimental designs (Denton, 2008; Shneyderman, 2006), but 
no experimental study was conducted prior to VSRII. Part IJ of this report provides a detailed 
description of the intervention and how it was planned to be implemented under VSRIL. 


Context 


To select the participating schools, VDOE initially reviewed results from the Virginia Standards 
of Learning (SOL) English/Reading assessments conducted in spring of 2007 and 2008. School 
divisions that had schools with large numbers of striving readers in seventh and eighth grade 
where then invited to participate. The three participating school divisions — Norfolk City Public 
Schools (Norfolk), Richmond City Public Schools (Richmond) and Roanoke City Public Schools 
(Roanoke) — are located in urban, high poverty settings. All participating schools serve students 
from grades 6 through 8. Two of the three participating school divisions, Norfolk and 
Richmond, had been part of the Reading First Initiative. Roanoke, although not involved in 
Reading First, expressed strong interest and commitment to the study when approached by 
VDOE. 


Norfolk City, the second largest independent city in Virginia, is located on the eastern side of the 
state and overlooks the Chesapeake Bay. The city is headquarters for the Norfolk Naval Base 
and the North Atlantic Treaty Organization (NATO) Allied Command Transformation. The 
school division had three of its seven middle schools participating in the project: Azalea Gardens 
Middle School, Blair Middle School, and Norview Middle School. In SY 2009-2010, the year 
prior to the VSRII implementation, Norfolk served 34,068 students in 51 elementary, middle and 
high schools. The student enrollment in Azalea Gardens totaled 832 students and the school was 
in Year 2 of school improvement, according to the NCLB accountability requirements. Blair 
Middle served 967 students while Norview Middle served 925 students. Both schools were in 
Year 4 of school improvement.” 


* Information on the school performance for this section was retrieved from the Virginia Department of Education, School Report 
Card, https://plpe.doe.virginia.gov/reportcard/; demographics were retrieved from the fall membership site, 
http://bi.vita.virginia.gov/doe_bi/rdPage.aspx?rdReport=Main&subRptName=Fallmembership 
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Richmond City, centrally located, is the state capital and the third largest city in Virginia. 
Lucille M. Brown Middle School was the only one of the eight Richmond City middle schools 
that participated in VSRII. In SY 2009-2010, the school division served 22,994 in 47 schools. 
Lucille M. Brown served 701 students and offered a Title I Schoolwide Program. The school 
had made AYP in SY 2009-2010. 


Roanoke City, located in the Blue Ridge Mountains, is a commercial hub for the southwest 
Virginia—southern West Virginia corridor. All five of its middle schools were part of VSRIL. 
In SY 2009-2010, the school division served 12,948 students in 25 schools. Lucy Addison 
Aerospace Magnet Middle School, which served 474 students, was a Title I — Targeted 
Assistance Program school in Year 2 of school improvement. James Breckinridge Middle 
School served 612 students while James Madison Middle School, served 593 students. Both 
schools were in Year 3 of school improvement. Stonewall Jackson Middle School, with 492 
students, offered a Title I Schoolwide Program, and made AYP that school year. Woodrow 
Wilson Middle School served 459 students and also made AYP. Table 1 summarizes the 
enrollment, minority status, and AYP status of participating schools in the fall of 2009-2010, the 
school year prior to the VSRII implementation. 


Table 1. Information on VSRII schools (school year 2009-2010) 
Student Enrollment 
Schools Minority NCLB Status 
(%) 
Azalea Gardens Year 2 of school improvement 
Norfolk City | Blair Year 4 of school improvement 
Norview Year 4 of school improvement 


Lucille M. Brown Made AYP 


School 
Division 


Richmond 
City 


Lucy Addison Aerospace Year 2 of school improvement 
James Breckinridge Year 3 of school improvement 
James Madison Year 3 of school improvement 
Stonewall Jackson Made AYP 
Woodrow Wilson Made AYP 


Roanoke 
City 


The next section provides a description of the intervention, as proposed by the publisher, and the 
adaptations that were made to the intervention to address the needs of each participating school 
during the VSRII project. 
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PART I: INTERVENTION AND LOGIC MODEL 


Description of the Intervention Model’ 
Classroom Intervention 


Published by Cambium Learning Group (Cambium), Passport Reading Journeys (PRJ) is an 
adolescent reading intervention that blends teacher-led targeted instruction with student-centered 
strategies, and uses information technology to engage student and reinforce instruction. The 
program is formatted as a series of lessons designed to be delivered over the course of one school 
year. Across grade levels, the intervention maintains the same structure but the content and 
reading level changes. The intervention is called PRJ Beginnings for sixth graders; PRJ I for 
seventh graders; and PRJ II for eighth graders. PRJ III focuses on struggling ninth grade 
readers. VSRII schools implemented PRJ J and PRJ II. 


The intervention encompasses daily, 50-minute lessons that provide explicit, systematic 
instruction in critical reading skills. The lessons are organized in Expeditions, for a total of 15 
Expeditions per grade level. Each Expedition is organized in ten-lesson routines to facilitate 
teacher-led instruction and students' independent practice. Lessons one, three, six, and eight of 
each Expedition are organized around whole-group instruction in which students are introduced 
to new vocabulary and a new reading passage. Lessons two, four, seven, and nine include 
whole-group review of the previous day’s instruction and the opportunity for students to re-read 
the passage to build fluency, independently or with a partner. During this period of independent 
or small-group structured practice, the interventionists are expected to work intensively with 
students who present specific needs. Lessons five and ten of the Expedition are spent in 
independent or paired practice on Strategic Online Learning Opportunities (SOLO). SOLO is an 
interactive, web-based reading resource component that provides students with opportunities to 
engage in self-paced practice of vocabulary and comprehension skills, and assess their learning. 


Core instructional elements in reading 


PRJ blends reading foundational skills, vocabulary instruction, direct and explicit comprehension 
strategies, text meaning and interpretation, and writing. The intervention is based on reading 
research and research in learning, including works from Baker, Simmons, & Kame’enui (2004), 
Beck, McKeown, & Kucan (2002), Biancarosa, & Snow (2006), Deshler, Palincsar, Biancarosa, 
& Nair (2007), Gersten, Fuchs, Williams, & Baker (2001), Graham & Perin (2007), Marzano 
(2004), Mastropieri, Scruggs, & Graetz (2003), Scammacca, Roberts, Vaughn, Edmonds, 
Wexler, Reutebuch, and Torgesen (2007), and Schatschneider, Buck, Torgesen, Wagner, 
Hassler, Hecht, & Powell-Smith (2004). 


Instruction in reading foundational skills is provided through the word study component of the 
Expeditions. The students with the lowest word reading ability are taught with a thirty-lesson 
word study program, beginning with a review of single letter-sound correspondences. These 
explicit word study lessons may be provided prior to implementing the first Expedition lesson or 


3 This sub-section was reviewed by Cambium Learning Group staff for accuracy. 
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on alternate days once the intervention sequence has begun. Explicit instruction is delivered 
with a focus on decoding, spelling, vocabulary, comprehension, and fluency. In addition, the 
students receive continued systematic and explicit instruction in practices that teach them to be 
flexible decoders. Lessons offer instruction in affixes, sight words, decoding multisyllabic 
words, spelling, and word or phrase fluency. 


Vocabulary instruction is addressed through the use of explicit instruction of word meanings and 
development of strategies to determine unknown words through morpheme analysis. A planned 
sequence of vocabulary skills and multiple exposures of high-utility words are meshed within the 
passages, comprehension activities, and text discussions. Affixes and roots are explicitly taught 
to students in a sequential pattern that is supported by the identified words in the passages. 
SOLO provides self-paced practice on vocabulary and comprehension skills. Multiple tools help 
students determine word meaning and contextual use in self-selected, Lexile-leveled* reading 
passages. New words are introduced with age-appropriate definitions and examples. Supports 
include automated clues or prompts and a function that allows students to click on difficult 
words to hear their pronunciation and definition. 


Direct and explicit comprehension strategies are woven into instruction to help students develop 
skills that are traditionally lacking among striving readers, such as making and confirming 
predictions, identifying or generating main ideas, summarizing, and making inferences 
(Baumann, Font, Edwards, & Boland, 2005). Comprehension skills are taught explicitly and 
applied to expository passages both in the text and in SOLO. The lessons incorporate strategies 
for making connections, asking questions, visualizing, and making inferences. Students also 
examine organizational text features that serve as frames for information and logical links 
between ideas. Comprehension strategies are scaffolded in three stages: interventionist 
modeling, interventionist assistance with student practice, and student independence. The stages 
represent a gradual shift in responsibility for learning from the interventionist to the students. 
Direct instruction includes modeling in which the interventionist reads aloud to show students 
how to use the reading strategies. A thinking aloud process is employed to make thought 
processes transparent to students. Modeling is followed with direct, guided practice and self- 
assessment to enable students to apply the newly learned skills and strategies in a variety of texts 
that cover varying levels of reading ability. 


Discussion of the text meaning and interpretation are elicited through questions posed by the 
interventionist during and after reading. In the first reading of the selection, the interventionist 
asks literal comprehension questions to ensure understanding and to model the metacognitive 
process of self-monitoring. After students complete their reading, the interventionist asks critical 
thinking questions that reflect the various levels of the revised Bloom’s Taxonomy (Anderson & 
Krathwohl, 2001). This interventionist-directed questioning is integrated with student—generated 
questions as a key reading comprehension strategy during reading and a way for students to 
monitor and deepen their understanding of the text. 


“ Lexile is a numeric representation of an individual’s reading ability or a text’s readability based on the work of Jackson Stenner, 
from MetaMetrics, Inc. 
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PRJ includes a two-fold approach to writing. One approach is writing in response to reading, 
which helps students check their understanding, reinforces returning to the text for more 
information, and sharpens critical thinking skills. Every Expedition integrates writing practice 
and instruction. The second writing approach is a writing lesson at the end of each Expedition 
that extends the comprehension skills and content into a writing topic. These lessons are 
designed to help students develop writing proficiency. Writing instruction includes a focus on 
generating ideas, elaboration, organization, word choice, sentence fluency, and conventions. 
Lessons employ explicit instruction, models of effective writing, and lesson-specific rubrics to 
enable self- and peer-evaluation. 


Motivation and engagement in literacy 


To improve student motivation and engagement in literacy learning, PRJ offers a library as part 
of its instructional materials. The primarily nonfiction texts have been field-tested for high 
interest with middle school students and reach across the curriculum to foster literacy 
development in social studies and science content areas. Examples of topics for the Expeditions 
include The Science of Catching Criminals, Predicting the Perfect Storm, The Internet: A Wired 
Word. The characters, content, and activities target students who represent diverse cultural and 
linguistic groups. DVD segments are presented before and after each Expedition to provide 
background knowledge and create the foundation for understanding of content. Each video 
segment is hosted by a teen who asks probing questions, highlights essential content-area 
vocabulary words, makes relevant connections to students’ lives and engages them in thinking 
about the topics at hand. 


Use of technology 


Technology is incorporated into PRJ through the SOLO component. In addition, lessons one, 
six, and nine include video technology in the form of DVD segments. SOLO is based on 
Computer Assisted Collaborative Strategic Reading (CACSR). Research has found that 
computer-assisted reading instruction helps struggling readers by providing individualized 
instruction, immediate feedback, a motivating learning environment, a way to monitor student 
progress, and a way to maintain student interest (Kim, 2002; Kim, Vaughn, Klingner, Woodrugg, 
Reutebuch & Kouzekanani, 2006). 


Professional Development 


Cambium Learning Group offers diverse professional development activities for the 
interventionists that include launch training, online product training, ongoing consultative 
support, coursework on adolescent literacy, and data analysis meetings. The launch training, the 
online product training, and the online support are part of the intervention’s package, while the 
other activities depend on separate contracts between the schools or school divisions and the 
developer. 
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Group professional development 


The launch training is a six-hour, two-day session that introduces the interventionists to PRJ. 
The objective of the launch training is to prepare the interventionists to implement the 
intervention with fidelity. Participants learn about the intervention, and are instructed in specific 
practices, such as administering the assessment measures, grouping students, setting up their 
classrooms, structuring small and large group instruction, and using all intervention materials. 
The training includes time for practice on lesson delivery, and instruction in a computer lab on 
SOLO, the technology components of the intervention. Training on Voyager data management 
system (VPORT) and classroom management are also included. 


During launch training, participants are invited to observe and reflect as the trainer demonstrates 
a lesson. Following the demonstration, participants have opportunities to practice teaching the 
lesson. They regroup at the end of the training to debrief and plan next steps. Materials include 
DVD footage of classroom instruction, illustrations of program features, and the measures to 
practice administering and scoring the assessments. Tutorial booklets introduce the key features 
and components of the program, present sample lessons at each grade level and a review of the 
assessment component, including entry points, and provide suggestions for managing time and 
working with students with special needs. 


The online training modules provide instruction in a self-paced, interactive environment that 
allows the interventionists to search, annotate, and bookmark information. Each module 
includes curriculum, assessment, and implementation overviews, and provides links to a library 
of video segments. The modules also offer suggestions on classroom management, and on 
understanding Lexile levels. At the conclusion of each section, the interventionists take a quiz to 
check the knowledge gained. They can redo the modules to improve knowledge or come back to 
them later to refresh information. 


Coursework on adolescent literacy are delivered through VoyagerU, Cambium’s professional 
development arm. Two 15-hour courses present foundational information about adolescent 
literacy, define the specific reading skills the students need in order to master each academic 
subject, and identify the best strategies to help middle school students develop their reading 
comprehension skills in these subjects. These courses were developed by Deborah Reed, 
principal investigator and project manager for the Texas Adolescent Literacy Academies, Diane 
Lapp, Distinguished Professor of Education, San Diego State University and a member of the 
International Reading Association Hall of Fame, and Douglas Fisher, professor of language and 
literacy education at San Diego State University and co-director for the Center for the 
Advancement of Reading at the California State University. Details about the professional 
development planned for VSRII are provided in the next section (VSRII Logic Model.) 


Individual supports 
Cambium offers individualized supports for teachers who are implementing PRJ through trained 
experts, the Voyager Implementation Specialist (VIS). The VIS visits each participating schools 


to observe how the intervention is being implemented. Debriefings are conducted with each 
interventionist, the building principal, and other designated parties. The frequency of visits is 
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dictated by interventionists’ needs, but also by the contract established between Cambium and 
the school or school division. The VIS review student data with the interventionist on an 
ongoing basis to accurately formulate prescriptive technical assistance, which must be geared 
towards each interventionist’s needs. The interventionists are also provided with the VIS’ phone 
number and e-mail address, and are encouraged to contact them as needed. 


Assessments 


The assessment system within PRJ includes benchmark assessments, fluency measures, end-of- 
lesson assessments, progress monitoring, and student self-assessments through SOLO. These 
assessments are based upon Lexiles to allow educators to quickly estimate expected reading 
comprehension and monitor progress. Lexile, developed by MetaMetrics, Inc., is a measure of 
the difficulty of comprehension of a text (Stenner, 2001; Stenner & Wright, 2004). The measure, 
based on calculations of word frequency and sentence length, is presented on a scale that ranges 
from OL to 2,000L. Text measures at or below OL (zero Lexiles), are reported as BR (Beginning 
Reader). 


The benchmark assessments were developed using the Rasch one-parameter item response 
theory model to relate a reader's ability with the difficulty of the items. The primary sources of 
validity evidence for Lexiles come from examining the content of the PRJ assessments and the 
degree to which the assessments measure reading comprehension (Lennon & Burdick, 2004). 
The Reading Benchmarks are expected to be administered in a whole group format three times 
per school year during specified periods to assess comprehension (MetaMetrics, 2009). The 
Reading Benchmark I, conducted at the beginning of the school year (September), is used to 
place students in the appropriate level of reading materials and in one of three appropriate levels 
of text in SOLO. The Reading Benchmarks II and III, conducted in January and May, are used to 
monitor student progress on vocabulary and comprehension. 


The Vital Indicators of Progress (VIP) measures identify students who have underlying decoding 
problems and who can benefit from targeted word study. VIP was developed by Roland Good 
and colleagues at the University of Oregon, and includes six tests: Letter Naming Fluency, Initial 
Sound Fluency, Phoneme Segmentation Fluency, Nonsense Word Fluency, Reading Connected 
Text, and Retell Fluency (Peyton & Macpherson, 2008). Only the Reading Connected Text 
assessment is used in PRJ J and // to identify students who would benefit from instruction in 
reading foundational skills. The test is administered three times per school year. 


Formative assessments are also conducted at the end of each two-week Expedition lesson series. 
These are criterion-referenced tests that measure comprehension and vocabulary skills that have 
been taught during the lesson series. Additionally, student self-assessments are available through 
the SOLO Progress Report. The SOLO reports provide students guided feedback on their reading 
speed and accuracy scores for comprehension as they progress in each Expedition. The 
interventionists can review the feedback provided to students by logging into VPORT. Based on 
student performance on these assessments, the interventionists are directed to re-teaching 
opportunities that are targeted to the specific skills where students have demonstrated difficulty. 


Virginia Striving Readers Intervention Initiative, Year One Implementation 9 


Target population 


PRJ I and IJ are geared to students in grades 7 and 8, respectively, who are defined as struggling 
readers by their schools. The intervention does not propose a maximum cut score for 
participation. Likewise, there is no established minimum cut score. The intervention 
incorporates a number of differentiating strategies that are designed to address students with a 
broad range of reading levels and students who have limited English proficiency. PRJ also 
addresses the needs of students with disabilities who are able to receive instruction in a 
classroom environment and can be served through group instruction. 


Desired characteristics of the interventionists 


PRJ reflects a prescriptive intervention. Each interventionist receives a teacher’s guide that 
includes an explanation of the intervention, the goals, scope and sequence of each component, 
followed by detailed guidelines on how the lesson must be taught. The interventionist is 
expected to follow the guidelines, and maintain the scope and sequence of each lesson’s 
components. Small variations within the lessons are allowed to address differences in class 
period and students’ needs, as explained below. 


Decisions about hiring interventionists are left to the local education agencies (LEAs). The 
intervention’s scripted format and the professional development offered are intended to facilitate 
instruction by experienced and non-experienced interventionists alike. For teachers who do not 
have a reading background, Cambium provides additional training on reading through its 
professional development branch (VoyagerU). 


Desired characteristics of the classrooms 


Cambium’s requirements for PRJ classrooms include appropriate space for small group 
instruction and for storage and use of material connected to the lessons, including teacher guide 
book, students’ workbook, and the library. Additionally, the classrooms should have a DVD 
projector and computers for SOLO lessons. Cambium recommends a maximum of 20 students 
per classrooms. 


Recommended intensity for the students 


PR/J is to be taught daily in a 50-minute period class within one school year. The intervention 
has been adapted for block time implementation (e.g. 90-minute period), as more schools adopt 
the longer periods of instruction. Interventionists are expected to cover one lesson per period or 
two shortened lessons in the 90-minute time. The pacing of the lesson should be a balance 
between the expected one lesson per period and students’ needs. If the lesson cannot be 
completed within the allotted period, the interventionists are instructed to continue it the 
following day, starting from the point they stopped the day before. Reducing writing time is an 
allowed strategy to accommodate the pacing, but reducing reading time is not recommended. 
The interventionists are not expected to complete all 15 Expeditions within the year, although 
they should try to cover as many as possible. PRJ students are assessed frequently for progress 
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on vocabulary, comprehension, and fluency, with the assessments described above, and results 
from the assessments are used to plan instruction. 


VSRITI Logic Model 


Overview 


Figure | displays a graphic representation of the logic model that informs VSRII. The model 
includes two components and one outcome. The components are the professional development 
model and the classroom instruction model. These two models follow as close as possible the 
model proposed by the PRJ developers, with a few variations that addressed the specific needs of 
participating schools. The expected outcome at the end of the initiative was that students in 
seventh and eighth grades who participated in the project read at least one grade level higher on 
standardized assessments and/or score proficient on the statewide assessment. 


The personnel resource available to support VSRII implementation included 23 educators 
located within the schools, LEAs, and the state education agency. Each participant had defined 
roles and responsibilities within the project and in relation to the study conducted by RMC, as 
explained below. 


At the state level, the Director of Elementary Instruction Services was responsible for overseeing 
the project and its interaction within VDOE’s overall strategic plan. Together with the Project 
Coordinator (henceforth called VSRII Coordinator), they worked in close collaboration with the 
VDOE’s Office of Middle and High School Education and Office of School Improvement to 
ensure that the intervention was aligned with the state standards and school improvement 
priorities. The VSRII Coordinator was responsible for the daily leadership of the project and 
held four major roles: (1) monitor the distribution of grant money to the schools and project- 
related materials and activities; (2) facilitate communication between the state, school divisions, 
schools, developers, and evaluators; (3) help participants to find solutions to potential challenges; 
and (4) support and monitor the project implementation. Two people held this position 
successfully. The first VSRII Coordinator had a background in elementary school reading and 
had been the coordinator for the Reading First Initiative. By the end of the planning year, he 
moved to another position within VDOE and was replaced by a new coordinator in July 2010. 
The second VSRII Coordinator, who had a title of Grants Specialist, was a reading coach with 
experience in middle schools. The project coordination represented a Full Time Equivalent 
(FTE) position paid with Striving Readers’ funds. Additionally, the grant paid a part-time Grant 
Manager who helped with the finances and purchase aspects of the project, including the 
disbursement of funds to the schools and consultants. 
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Figure 1. Logic model - Virginia Striving Readers Intervention Initiative (VSRID) 


Strategies (Model): Passport Reading Journeys I and IT 
Professional Development Model Classroom Model [7 and 8" grade classrooms] 


Year 2: Structure 
“*Summer 2010: Launch training (16 “* Year-long, daily 50-minute lessons (adapted also to 90-minute lessons) 
hours) “*15 Expeditions; each Expedition divided into 10 lessons: 
“School year: Lessons 1, 3, 6 and 8 (whole group) 
o In-school coaching (50 hours) (a) Advanced Word Study (Introduce Phonic Element and Sight Words) 
o Statewide data meeting (5 hours) (b) Before Reading (DVD/Video segment introduces comprehension, strategy 
o VoyagerU Adolescent literacy and vocabulary) 
vocabulary and comprehension (30 (c) Reading 
hours) (d) After Reading= comprehension check 
o Online modules (16 hours, optional) Lessons 2, 4, 7 and 9 
o Statewide year-end meeting (8 (a) Before Reading = review previous lesson instruction 
hours) (b) Reading = re-read previous passage 
Years 3-4: (c) After Reading = comprehension and vocabulary guided practice 
“* New teachers = as above (d) PRJ Library (independent practice) 
“* Returning teachers = same options (e) Word Study (small group) 
Attendees: Lesson 5 — SOLO (online independent application of reading strategies) 
“* Interventionists = mandatory Lesson 10 (2 days) - SOLO and re-teaching (teacher-selected activities) 
participation Content 
«+ Project leadership = suggested, but “Expeditions focus on themes related to science and social studies 
participation not required “Content is similar to grades 7 and 8, level of difficulty changes 
Provider: Assessment 
“* 3 Voyager Implementation Specialists “*Reading Benchmarks I - placement (September) 
(Under guidance of the Vice President “*Benchmarks II and III — progress on fluency(January and May) 
for Implementation Services for the “*Reading Connected Text — progress on comprehension (Jan and May) 
Southeastern Region) “*End of Expeditions — vocabulary and comprehension 
**SOL.O — self assessment 


7" and 8" grade 
students read at 
least one grade 
level higher on 
standardized 
assessments or 
score proficient 
on the statewide 


Personnel 


e VSRI Coordinator 
9 FTE interventionists <—— 9 principals (supervision and support) <——— 3 School Division Liaisons <——__» (5 FTEGrant Manager 


Classroom School building LEA VDOE 


Technology and supplies 
“* Schools: computer mobile-station; library 
“* Voyager : books (high-quality, high interest, leveled by reader ability) in print, audio and e-books 


RESOURCESS 


Teacher guide; student workbook; other supporting materials 
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Each of the three participating school divisions identified a LEA liaison who worked with the 
VSRII Coordinator, the school administrators, and PRJ developers to ensure the successful 
implementation of the intervention. The LEA liaison (1) monitored the use of grant money by 
the schools; (2) supported the school principals in the hiring and supervision of interventionists; 
(3) received reports from the V/S regarding the implementation of the intervention at each 
school; and (4) helped developers and principals find strategies to overcome challenges to the 
implementation. He/she was also the contact person for the implementation and impact study 
conducted by RMC. In this role, the LEA liaison facilitated the evaluators’ entrance into the 
schools and was responsible for the provision of student demographic and assessment data to the 
evaluators. The liaison for Roanoke City was the English Language Arts Supervisor. The 
liaison for Norfolk was the Senior Coordinator of Instruction, English. Richmond divided the 
responsibilities among two representatives. One liaison, the Title I Reading Instructional 
Specialist, was responsible for issues related to the intervention and its implementation in the 
classroom; the second liaison, the Grant Manager, was the point of contact for finances and data. 


At each of the nine schools, the principal or a designated representative assumed the school 
liaison role. Their responsibilities regarding the implementation included (1) hiring and 
supervising the interventionists as members of the school teaching staff; (2) monitoring the 
interventionists’ attendance at required state and PR/J training sessions; (3) ensuring that the 
intervention classrooms were adequately equipped; and (4) acting on formative feedback 
received from the VIS regarding the implementation of the intervention within the schools. 


For Cambium, the Vice President of Implementation Services in the Southeastern Region was 
responsible for overseeing the implementation at the VSRII schools. The Vice President 
supervised a team of three VIS assigned to the project (one per school division). At the assigned 
school, each VIS was responsible for (1) providing professional development and coaching for 
the interventionists; (2) supervising the work of the interventionists at each school; and (3) 
communicating their findings to the interventionists, the school principals and to the Vice 
President. 


Grant funds were used to pay the interventionists, the VSRII Coordinator and VDOE Grant 
Manager, purchase computers, DVD projectors, computer mobile station, and support material 
that were needed to equip the PRJ classrooms. The contract with Cambium included 
professional development, individual supports through a coaching system, and the provision of 
materials, such as guidebooks, workbooks, classroom library, and DVD library. 


RMC Research Corporation (RMC) was contracted by VDOE to conduct the implementation and 
impact study. The RMC team worked with the VSRII Coordinator, the school division liaisons, 
school administrators, and interventionists to plan the study and ensure its integrity. On the next 
pages, Figure 2 displays the VSR/J management chart for the implementation, while Figure 3 
displays the intersection of the implementation and impact study within the VSRII organization. 
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Figure 2. VSRII — management chart 
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Figure 3. VSRII - organization chart 
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VSRII professional development model 


Group professional development 


Table 2 summarizes the professional development activities that were planned for the 
interventionists and key VSRII participants for the first implementation year. 


Format 


Whole 
group, 
face-to- 
face 


Table 2. VSRII — planned professional development activities 
(summer 2010 — spring 2011) 


Trainer 


Cambium Learning 
Group (Vice President 
of Implementation 
Services and Voyager 
Implementation 
Specialists) 


Content 


Launch Training, including: 
administering assessment 
measures, grouping students, 
classroom set up, use of 
curriculum materials, practice 
lesson delivery, SOLO 
instruction, pacing, use of the 
Voyager data management 
system (VPORT) 


Interventionists, 
VSRII Coordinator, 
school division 
liaisons, school 
principals and two 
members of the 
evaluation team. 


Online, 
individual 


Cambium Learning 
Group (Modules 
created by Cambium 
consultants) 


Two 15-hour online modules on 
research on adolescent literacy 
focusing on vocabulary and 
comprehension for middle 
school students 


Interventionists 


Whole 
group, 
face-to- 
face 


Cambium Learning 
Group (Vice President 
of Implementation 
Services) 


Instruction on how to interpret 
student achievement data and 
use it to tailor instruction in the 
classroom 


Interventionists, 
VSRI Coordinator, 
principals, school 
division liaisons 


Whole 


group, 
face-to- 
face 


Cambium Learning 
Group 


Networking, information 
sharing, exchanging experiences 
(successes and challenges), 
reviewing assessment data and 
program updates. 


Interventionists, 
VSRII Coordinator, 
school division 
liaisons, school 
principals 


Required, intervention-related, total hours: 


Whole 
group 
(required; 
study- 
related) 


VSRII Coordinator 


Startup meeting: Introduce 
participants, clarify 
expectations, roles and 
responsibilities, and introduce 
the study. 


Interventionists, 
VDOE staff, school 
division liaisons, 
school 
administrators, 
evaluators 


Online, 
self-paced, 
individual 
modules 
(optional) 


Cambium Learning 
Group (Modules 
created by Cambium 
consultants) 


Modules that supplement launch 
training, including: curriculum, 
assessment, implementation, 
classroom management, 
understanding Lexile levels, and 
other topics (modules are being 
expanded); include video of 
model lessons. 
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Interventionists 


16 


VSRII incorporated a total of 59 hours of required, intervention-related professional 
development for the interventionists. The required professional development included 16 
hours of launch training, 30 hours of online modules on adolescent literacy, 5 hours on 
interpreting the formative assessment data, and 8 hours on revising and reflecting on the 
lessons learned during the first implementation year, and preparing for the second year. The 
interventionists were also required to participate in 4 hours of a statewide, startup meeting with 
the school principals, school division liaisons, VDOE staff, and evaluators that introduced all 
participants to the implementation and impact study. 


The professional development on adolescent literacy comprised two 15-hour courses offered 
through VoyagerU. The courses provided information on the basic literacy skills that are 
missing for adolescent struggling readers and on interventions that research has shown to be 
successful to build those skills. Another 16 hours of online professional development were 
available to the interventionists as needed. The optional modules are intended to reinforce or 
clarify the topics discussed during the launch training or coaching sessions. 


As faculty members within their schools and school divisions, the interventionists were also 
required to join the professional development days offered to all faculty members. These 
activities were not related to intervention and, therefore, were not included in the logic model. 


The group professional development activities were available to the VSRII Coordinator, school 
division liaisons, and school principals or representatives. Attendance for this group was not 
required, except for the startup meeting and the meeting planned for the end of the first 
implementation year. The evaluator team was represented in the launch training by two 
reading specialists who led the site visits. 


Individual supports 


VDOE contracted Cambium Learning Group to provide face-to-face individualized coaching 
supports for each interventionist. Three VIS were identified, one for each school division, and 
worked under the supervision of Cambium’s Vice President for Implementation Services in the 
Southern Region. The VIS worked with the interventionist for a full day during each visit, but 
the number of days they were expected to provide coaching was negotiated a priori with the 
school divisions. VDOE budgeted for a total of ten full-days of on-site coaching for each 
interventionist. As the implementation started, the full coaching day was established as an 
average of five hours per visit, with a maximum of 50 hours per implementation year. The VIS 
was also available to address questions and concerns as needed via conference call and e-mail. 


The topic of the individual on-site coaching support was tailored to the interventionists’ needs, 
but the coaching model followed a similar format in each school. First, the VIS observed a 
lesson taught by the interventionist. The VIS would then model a lesson, followed by a 
debriefing session during which the VIS discussed the observation with the interventionist and 
made recommendations for improvement. After that, the VIS observed the interventionist 
teaching another lesson on a different day to see if he or she was implementing the 
recommendations. During the coaching visits, each VIS used the Cambium Learning Group’s 
Index of Fidelity of Implementation (IFI) to assess how close the intervention implemented in 
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the classroom was to the model intervention. The IFI was used as a foundation for the 
observation instrument used by the evaluators during the site visits. 


The first VSRII Coordinator decided to leave the coaching process to the VIS, but the second 
VSRII Coordinator, who started in the summer of 2010, planned monthly visits to the schools 
as a second line of face-to-face support to the interventionists. The visits would be used to 
observe the interventionists, provide feedback, and maintain communication with the school 
principals. As faculty members, the interventionists were directly supervised by the school 
principal or a designated assistant principal. Yet, this supervision and supports were related to 
topics outside the intervention, such as understanding school regulations and expectations, 
ensuring that students abided by the school’s code of conduct and similar topics. The school 
division liaisons were also planning to visit the classrooms to follow through on 
recommendations made by the VIS and ensure that the implementation was progressing 
without problems. 


Planned classroom instruction 
Adaptations to the classroom model 


The VSRII planned classroom model followed the developer’s model, as described in the 
previous section. Adaptations were discussed during the planning year to address the different 
class periods at the participant schools. Two schools were using block scheduling with 90- 
minute classes. For these schools, Cambium suggested that interventionists present two 
lessons in one day, reducing writing time to adapt to a 45-minute lesson rather than 50 minutes. 
For schools with class periods of less than 50 minutes, the proposal was to shorten writing time 
and continue the lesson the following day. Another strategy to balance pacing included using 
an extra day in the final Expedition lesson (Lesson 10) to complete instruction or re-teaching. 
Cambium proposed a menu of options that interventionists could use for this extra day. These 
options addressed the possibility that some interventionists would need fewer re-teaching 
strategies and could use the time for enrichment. Alternatively, the menu also contained 
options for students who required further support. While writing time could be curtailed or 
even eliminated to maintain the pacing, the developers reinforced the need to maintain time 
reserved for reading, the sequence of the different components of the lessons, and the overall 
structure of the Expeditions. Cambium specialists were working on adapting some of the 
presentations to use with interactive white boards, which are commonly present in Virginia’s 
classrooms. 


Experiences for control group students 


The school year in Virginia extends for approximately 180 days. Each school planned for 
daily lessons with seventh and eighth grade students being taught separately. The classes were 
planned to coincide with time reserved for electives to ensure that students participating in the 
study could attend the core content area classes offered to all students in the same grade level. 
The school division liaisons assumed the commitment that control group students would not 
receive PRJ or other supplemental reading instruction. 
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Roanoke and Norfolk planned to invite students in the control group to join elective classes 
that included arts, foreign languages, physical education (in Roanoke), and expansions of core 
content area classes, such as literature. Richmond was scheduling PRJ for fifth period and 
students in the control group would be offered enrichment classes that were not reading 
supplemental. It is of note that Richmond schools were providing instruction with PRJ for all 
its middle school students. At Lucille M. Brown, PRJ was provided to sixth graders and 
students in grades 7 and 8 who were not eligible for Striving Readers, such as those who 
scored one year below grade level on the GMRT or, although proficient on the statewide 
assessment, were close to the cut score. 


Remediation classes 


By law, Virginia schools are required to provide supports to all students who do not attain 
proficiency on the statewide assessments. Most schools offer supports during regular English 
classes, through small group instruction, but remediation classes are also offered after school, 
on Saturdays, and during summer vacation. These supports are offered to all students in need. 
The remediation classes focus on providing students with better understanding of the standards 
and skills to succeed in the state assessment. The classes are held by the regular school faculty. 


After a lengthy discussion among developers, VSRII key participants and the evaluators, it was 
agreed that these supports did not represent supplemental reading instruction, were provided to 
all students eligible to remedial education (therefore, treatment and control group students had 

equal chance to participate), and did not involve the interventionists. Therefore, they were not 

considered to be in conflict with the Striving Readers’ requirements. 


Characteristics of interventionists 


A total of nine teachers were hired to provide instruction in PRJ J and II. Initially, two schools 
in Roanoke planned to share one Full Time Equivalent teacher. Eventually, with support from 
VDOE, the school division decided to hire 4.5 FTE interventionists; with one interventionist 
per school to facilitate scheduling (the smallest school had a 0.5 FTE). 


The hiring of the interventionists followed the process used by the school divisions to hire their 
regular teaching staff. The position was announced on the school divisions’ web sites and in 
local newspapers, and the applicants were interviewed by a panel that included the school 
principals, who made the final decision. The teachers were required to have a valid Virginia 
teaching license with an elementary or middle education endorsement that included grades 
seven and eight. No reading certification or endorsement was required, although two of the 
interventionists had reading certification. Four of the schools hired existing faculty for the 
position. Five other interventionists were new to the schools; of these, only one was a newly 
graduated teacher. 


The teachers were required to dedicate at least 80 percent of their time to the intervention, 
while the remaining 20 percent was divided between planning time and administrative duties. 
Some of the activities that the interventionists were involved included cafeteria and hall duty 
during the beginning and end of the day and during lunch time. 
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Classroom characteristics 


Assigning permanent classrooms to the interventionists was the ideal proposal, as they could 
leave the intervention material in the classroom. However, the schools were facing 
overcrowding and accommodations were a challenging process, as explained in Part II. All 
participating schools had computer laboratories that were used for SOLO. Additionally, 
VDOE allocated funds in each school budget for the purchase of software and hardware, 
including 14 computers and a DVD projector per school. Since the principals could not assure 
that the interventionists would be assigned permanent classrooms for the year, VDOE allocated 
money for the purchase of carts that could transport computers, DVDs, and the PRJ library 
from a secure location to the classroom, if needed. Money was also planned for additional 
computers to be purchased during the implementation years two and three. The schools 
planned to maintain the available slots per classroom to not exceed 15 students. 


Assessments 


As described in Part I (p. 9), PRJ uses a number of assessments to identify students’ needs and 
inform instruction. Conducting, analyzing and using the assessments to manage student 
learning were part of the launch training and the individual support that each VIS provided to 
the interventionists. A full-day workshop was also planned for the middle of the year to 
refresh information about data analysis from these assessments. 


VSRIL used the Group Reading Assessment and Diagnostic Evaluation (GRADE) as a 
diagnostic tool to identify reading skills that students have mastered and those where they 
needed intervention. Published by Pearson Learning Group, GRADE is a norm-referenced test 
that can be group administered. The test has eleven levels and can be applied to grades K-12; 
M is the level corresponding to middle school. Subtest and composite scores can be converted 
to stanines, standard scores, percentiles, normal curves equivalents (NCE), and grade 
equivalents. Reliability coefficients for alternate form and test-retest were in the 0.90 range. 
Concurrent and predictive validity was assessed with other standardized reading assessments, 
such as Terra Nova and Iowa Test of Basic Skills. GRADE was administered to the students in 
the treatment group in early September, and information was used to help the interventionists 
make decisions about reading levels for each treatment student. 


The impact study employed two assessments that were used for determining eligibility for the 
study and assessing outcomes. These assessments were the Gates-MacGinitie Reading Test 
(GMRT), 4" Edition, and the Virginia Standards of Learning (SOL) English/Reading 
assessment. The SOL tests were developed by a testing contractor under the guidance of 
VDOE staff. Statistics were established for all grade levels based on samples of 8,000 students 
or more and included traditional difficulties (p-values), item-option response distributions (all 
respondents, high-, middle- and low-ability; gender and ethnic groups); and biserial and point- 
biserial correlations (Virginia Department of Education, 2005). Results for the 
English/Reading test are presented as raw scores, scaled scores, and performance levels (below 
basic, basic, proficient, and advanced). SOL results are used to assess AYP. 
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The psychometric qualities of the GMRT 4"" Edition, including evidence of reliability and 
validity, are well documented in a 2002 technical report (MacGinitie, MacGinitie, Maria & 
Dryer, 2002). Alternate form reliability coefficients for test levels appropriate for grades 6-8 
range from 0.82 to 0.91. K-R 20 estimates of reliability range from 0.90 to 0.95. The GMRT 
has been normed for both fall and spring administration and a variety of score types are 
available to support analysis and interpretation (e.g., extended scale scores, percentiles, 
stanines, and grade equivalents). The test was renormed during SY 2005-2006 (Maria & 
Hughes, 2008). The school divisions administer the GMRT in April and the SOL in May of 
each school year. 


Study eligibility was assessed with results from the GMRT and SOL English/Reading 
administered to students in grades 6 and 7, while outcomes were measured by GMRT and SOL 
results for students in grades 7 and 8. Table 3 summarizes the assessments used in VSRII. 


Instruction 


Table 3. VSRII - list of assessments 


Application 


Reading 
Diagnostic 


Group Reading Assessment 
and Diagnostic Evaluation 
(GRADE) 


Twice per year (fall and 
spring semesters) 


Diagnose students’ 
reading needs 


Placement 
and 
Monitoring 


Impact study 


Reading Benchmark I 


September 


Place students at 
appropriate reading 
levels 


Reading Benchmarks 
II and II 


January and May 


Monitor student 
progress on vocabulary 
and comprehension 


VIP Reading 
Connected Text (RCT) 


Ongoing 


Monitor student 
progress on fluency 


Eligibility 


Virginia Standards of Learning 
(SOL) — grade 6 & 7 


Spring of the previous 
school year (generally 
May) 


Gates-MacGinitie Reading 
Tests (GMRT) 4th Edition — 
grades 6 and 7 


May of the previous 
school year 


Identify struggling 
readers 


Outcomes 


Virginia Standards of 
Learning (SOL) — grades 7 & 8 


May of the school year 
being studied 


Gates-MacGinitie Reading 
Tests (GMRT) 4th Edition — 
grades 7 & 8 


VSRII eligible students 


April of the school year 
being studied 


Impact evaluation 


VSRII served students in seventh and eighth grades at nine middle schools located in three 
school divisions in Virginia, as listed in the Introduction. For the rest of this report, the schools 
will be named by numbers, randomly attributed, to maintain their privacy. In SY 2008-2009, 
these nine schools served a total of 5,492 students in grades six through eight. Students 
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eligible for the free and reduced meals (FARM) program comprised 61 percent of the schools’ 
population. Six of the nine VSRII schools did not make AYP in the SY 2008-2009. 

In spring 2010, the schools administered the GMRT 4" Edition to all students in sixth and 
seventh grades. Students whose test results were equivalent to two or more years below their 
grade level were eligible for VSRII. Additionally, students in sixth and seventh grades who 
did not pass the spring 2010 SOL English/Reading assessment were also eligible for the 
intervention regardless of their GMRT scores. This process would have been repeated in 
spring 2011 and spring 2012 with all sixth and seventh grade students had the VSRII project 
continued. 


Two exclusion criteria were adopted for VSRII eligibility: students with an Individual 
Education Plan (IEP) that precluded their participation in the study, and students whose parents 
requested their children be exempt from the study. The processes used to determine eligibility 
and randomly assign students to the treatment and control group are detailed in Part III. 


Expected student outcomes 


VDOE proposed two outcomes for the project: (a) all students receiving the intervention would 
improve their reading skills by a minimum of one grade level or its equivalent, as measured by 
the GMRT; and (b) at least 50 percent of students receiving the intervention would score at or 
above the proficiency level on the Standards of Learning English/Reading. 


In conversations with the VSRII Coordinator, school division liaisons, and school principals, 
the evaluators offered to measure other student outcomes, such as motivation or changes in 
reading strategies. The participants decided to focus solely on assessing student reading levels 
through standardized tests to minimize interference in the schools’ daily routines. 


Planning Year (2009-2010) 
Preparing for the implementation 


As SY 2009-2010 started, the VSRII Coordinator established a schedule of monthly phone 
calls with the school division liaisons, school principals, representatives from Cambium 
research and implementation divisions, and the evaluators. The calls were used to clarify 
questions regarding the intervention and the study, finalize the selection of schools, and define 
a Memorandum of Assurance (MOA) between VDOE and the school divisions. The MOAs 
outlined responsibilities for each VSRII school, and were signed by the school principals and 
school division superintendents. During the months that extended from September and 
December 2009, two schools dropped out of the project due to conflicting requirements from 
another grant they were receiving, and Roanoke added one more school, to include all its 
middle schools in the study. USED approved the changes, which did not impact the potential 
number of students in the project.’ 


> Table 1, in the Introduction section, displays the final list of participant schools. 
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Two statewide meetings were planned to bring all VSRII participants together. The first 
meeting was held on January 20, 2010. Present at the meeting were VDOE Director of 
Elementary Instruction Services, VSRII Coordinator, VDOE Grant Manager, LEA liaisons, 
principals for the nine participating schools, and evaluation team members. During the 
meeting, the VSRII Coordinator clarified the state management timeline, while the school 
divisions offered a brief presentation on their plans regarding the intervention and options for 
control group students. 


The hiring process for the interventionists started around May 2010 and followed the process 
used for the hiring of regular school faculty members. By the beginning of summer, nine 
interventionists had been hired, one for each school, reflecting 8.5 FTE. 


The second statewide meeting was held on August 11, 2010, following the PRJ launch 
training, and was attended by the VDOE Director of Elementary Instruction Services, the 
VSRII Coordinator, the VDOE Grant Manager, the LEA liaisons, the school principals, the 
interventionists, the evaluation team, the three VIS and their supervisor, and Cambium’s Senior 
Research Analyst. The meeting focused on the implementation timeline, the Voyager coaching 
activities, and the final details of the implementation and impact study. 


Preparing for the study 


RMC evaluation team was a constant presence in all phone calls and statewide meetings held 
by VDOE. RMC used the monthly conference calls to listen to the school division and school 
representatives about their previous experiences with similar grants, their concerns related to 
the study, and their resources to deal with the study requirements. They also used the calls to 
become familiar with the school divisions’ scheduling and regulations as they prepared to 
finalize the evaluation design. 


During the January 2010 meeting, the evaluation team presented the evaluation design, detailed 
the random assignment process, discussed the data requirements for the implementation and 
impact study, made suggestions on obtaining parental consent, and opened the forum for 
questions and answers. The school principals shared their concerns on how to deal with the 
randomization and explain to parents that control group students would be denied the 
intervention. Roanoke’s representative explained that buy-in was facilitated by the fact that the 
intervention was limited to one year and that eligible students could be assigned to the 
intervention the following year. The other two divisions were grasping with similar concerns 
of buy-in within the schools. Another concern shared during the meeting was related to 
scheduling, as the intervention would be a new course that had to fit within an already tight 
schedule, without sacrificing the other courses. School principals also raised concerns about 
space. The schools were overcrowded, they argued, and they had no rooms to accommodate 
extra classes. All principals were committed to finding a solution that avoided carting PRJ 
materials and computers from one room to the other. 


At the end of the meeting, three decisions were made. First, VDOE’s secure data transfer site, 
Single Sign-on for Web Systems (SSWS), was chosen as the locus for exchanging of student- 
level data between the schools and the evaluators, with VDOE providing the evaluation team 
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leader with access to the site. The second decision was related to the liaison staff between the 
schools and the evaluators for all data exchange activities. The school division liaisons 
accepted this responsibility. The final decision was related to parental consent. It was agreed 
that school principals would use their LEA procedures to communicate with the parents and 
allow them to exclude the child from the study (opt-out). The evaluators offered to help them 
through the process while not imposing their own procedures. 


In April and May 2010, the evaluation project director and the evaluators responsible for the 
random assignment process visited all nine VSRII participant schools. The evaluators 
proposed to meet with the whole school staff, if needed. In Roanoke, the evaluators were 
accompanied by the school division liaison and met with principals, assistant principals, and, in 
some schools, with members of the English Department. The evaluators also met with the data 
officer for the school division to finalize the data needs and transfer process. In Richmond, the 
evaluation meeting was held with the Assistant Superintendent, the school division liaisons, 
and the school principal. In Norfolk, the evaluators met with the school district liaison, the 
principals from two of the schools, and the assistant principal from the third school. The 
evaluators used these visits to explain the random assignment process once more, finalize 
decisions regarding parental consent and data transfer, and answer any questions that the 
school administration and faculty still had regarding the study. 


The start-up meeting in August 11, 2010 was the last planning activity and the first that 
included the interventionists. At that time, the evaluators answered final questions regarding 
the study and explained to the participants the meaning of the firewall between implementers 
and evaluators requested by USED. A level of trust had been established at this point and the 
school division liaisons and school principals expressed being comfortable with the study and 
the relationships with the evaluators. 
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PART II: IMPLEMENTATION STUDY 


Study Design 
Overview 


The evaluation of the VSRII implementation focused on assessing how closely the 
implemented intervention was to the proposed model. This information was to be used to 
further the understanding of the findings from the impact study. 


As described in Part I, PRJ is a highly-structured intervention in which the interventionists are 
expected to attend the required professional development and follow a script that details what, 
how, and when they will teach. Diversions from the model are not expected or welcomed, 
except for minor adaptations to adjust the required pacing within allocated classroom time. 

The intervention’s structure provided the framework upon which the research questions and the 
development of instruments for data collection were built. 


The evaluation strived to be as unobtrusive as possible to reduce the possibility of a 
“Hawthorne effect” (Gillespie, 1991). This effort was facilitated by the fact that Virginia 
school divisions already place students in remedial or enrichment classes in response to student 
academic indicators. Additionally, the students are tested with the two assessments that were 
used in the impact study. Moreover, the evaluators had no direct contact with the students 
involved in the project, except for silent classroom observations to which students are 
accustomed as a part of regular administrative visits. 


Research questions 


The implementation evaluation focused on the fidelity to the intervention model that had been 
proposed by the developers and incorporated in the VSRII Logic Model. The study was 
guided by two sets of questions. The first set explored the implementation of the intervention 
model. A second group of questions investigated factors that influenced the implementation. 


e To what extent did the PRJ implementation in the VSRI/ schools reproduce the 
proposed model? 

a. What types and how many hours of professional development were offered to 
the interventionists? 

b. What types and how many hours of professional development did the 
interventionists attend? 

c. How many hours of support did the Voyager Implementation Specialists (VIS) 
offer to the interventionists? 

d. What types of support did the VIS provide (e.g. lesson modeling, feedback) to 
the interventionists? 

e. How many hours of classroom instruction were planned? 

f. How many hours were provided? 
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g. To what extent did the interventionists follow the intervention’s structure 
(sequence of topics and allocated time)? 
h. To what extent did the interventionists use assessments to inform instruction, as 
prescribed? 
e What factors influenced the PRJ implementation? 
a. What factors facilitated the implementation? 
b. What factors created barriers to the implementation? 


Data Collection Plan 
Professional development model 


To document attendance to the professional development activities, the evaluators collected 
data from the following key informers: 


e Trainers — Cambium maintains logs with the types and hours of professional 
development offered, who attended, and what was covered during the activities. These 
data were accessed through VPORT. The evaluators also interviewed the VIS to 
obtain information regarding the coaching services provided and their perceptions 
about the PRJ implementation in the different schools, including the barriers and the 
facilitators encountered. 

e Trainees — The evaluators scheduled monthly check-ins with the interventionists to 
obtain their perspectives on project implementation. Due to difficulties contacting all 
of the interventionists by telephone on a monthly basis, beginning in early 2011 the 
interventionists could choose between telephone and online check-ins. A total of six 
check-ins should have been completed, but despite efforts, the number of check-ins 
varied with four interventionists completing all six check-ins, three completing five, 
one completing four, and one with only three check-ins. The check-in was 
complemented with short interviews conducted during the site visits. 

e VSRII Project Staff — The evaluators also collected information about participation in 
PD activities and coaching from the VSRII Coordinator and school division liaisons in 
each of the three districts. Additionally, the evaluators participated in regularly 
scheduled phone conferences with the VSRII project staff. 


Classroom model 


Data for the evaluation of the level of implementation of the PRJ classroom model were 
collected from the following sources: 


e Interventionists — As mentioned above, the evaluators scheduled regular phone or 
online check-ins with the interventionists to collect data on professional development, 
but also on classroom implementation, including scheduling of classes, planning time, 
other assigned duties, and reflections about the intervention and its implementation. 

e Voyager Implementation Specialists — The evaluators interviewed the three VIS to 
obtain information regarding their work with the interventionists and their perceptions 
of the barriers to or facilitators of project implementation. 
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e VSRII staff — The school division liaisons and VSRII Coordinator were also contacted 
regarding their assessment of how the implementation was occurring from the point of 
view of the school divisions and VDOE. 

e Classroom visits — Two evaluation teams observed the intervention classrooms, with 
each school being visited by one team. Each team was comprised of an evaluator with 
a reading background and an evaluator with a methods background. Four visits were 
initially planned: a month from the start of classes (October), middle of the winter 
semester (January or February), beginning of the spring semester (April), and end of 
the school year (end of May). 


The evaluators took extensive notes during the interviews. The interview data were then 
analyzed using categorizing and connecting qualitative data analysis (Maxwell & Miller, 
2008), and coded thematically to reflect the PRJ components highlighted by the developer in 
its Index of Fidelity of Implementation (IF). These components included: amount of 
instruction, quality of instruction, classroom management, use of assessment, and 
differentiation. Information from the interviews was used to foster understanding of the 
process of implementation in the different schools from the different actors’ perspectives, as 
well as to provide information regarding the barriers and facilitators to implementation. The 
check-in questionnaires, classroom observation rubrics, and interview protocols are included in 
Appendix B. 


Initially, the evaluators planned to interview the building administrators that had been assigned 
to supervise the interventionists at each school. However, during the planning year, it was 
clear that building administrators’ relationship with the interventionist was not directly related 
to the study and, attending their request, interviews were not conducted with them. 


The two of the evaluators who are reading specialists attended the launch training to become 
familiarized with the intervention. They then trained the other members of the evaluation 
team. They were also instrumental in developing the rubrics used for the site visits and 
supporting decisions regarding data to be collected and analyzed. Although the evaluators 
wanted to make unannounced visits, the complexity of school scheduling and the varying 
nature of the PRJ instruction made that impossible. To organize the visits, the evaluators 
obtained schedules from each school highlighting the weeks when the schools would be 
occupied with statewide tests, spring break and/or other events that disrupt regular classroom 
instruction. Additionally, they coordinated with the interventionists to avoid visits on days 
when the lesson time was used for assessments or independent work on the computer. The 
visit schedule was also coordinated with Cambium and VDOE so that evaluators were not 
present at the same time as the VIS or the VSRII Coordinator. 


The evaluators conducted two visits to each of the VSRII schools as planned—one in October 
2010 and the other in February 201 1—for a total of 47 observations. With the abrupt 
cancellation of the Striving Readers grant, and the need to maintain funding for the data 
analysis, the two final visits were cancelled. 
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Defining fidelity of implementation 
Professional development (PD) model 


Interventionists’ attendance to launch training, data training, and adolescent reading modules 
was required and facilitated through a number of measures. First, the interventionists were 
paid stipends for attendance to PD, including the online modules. Secondly, they had the 
opportunity to complete the 30 hours of online PD at a time that was convenient for them. 
Additionally, if they could not attend a scheduled training, the VIS provided the information at 
a more convenient time. Statements on the quality of the training were deemed unnecessary 
due to Cambium’s quality control measures. The face-to-face PD activities were conducted by 
the developers’ trained experts, and observed by a Cambium supervisor and their senior 
research staff. The online modules were created by reading experts and maintained by 
Cambium’s research division. 


Prior to entering the data collected into the PD database, the evaluators made attempts to 
resolve discrepancies between the hours found in the VPORT logs and those provided by the 
interventionists through conversations with the different groups of key informers. The 
evaluators then created a spreadsheet that included the types and hours of PD and support 
activities offered and the types and hours attended for each interventionist. As mentioned 
previously, the interventionists were required to attend 59 PD hours in the first implementation 
year. However, with the abrupt cancellation of the grant, the final statewide meeting was 
cancelled, thus bringing the required PD to 51 hours. Of these, 30 hours were offered through 
Web-based, asynchronous modules. 


While PD activities were required, coaching hours depended on the willingness and need of the 
interventionist as well as the availability of the VIS coach. Therefore, the score for the fidelity 
of implementation of the professional development model was calculated as the number of 
total PD hours attended relative to the number of required PD hours. The index was computed 
by school, since each school had one interventionist, and although they were teaching two 
different grades, the PD was the same. For instance, if an interventionist attended 67 hours of 
professional development out of 51 required hours, the interventionist/ school score was 1.31. 
The evaluators established a score of 1.00 or above as adequate fidelity to the professional 
development model. Scores below 1.00 were considered inadequate. The assessment of 
fidelity of PD implementation was limited to the interventionists. VDOE staff, school division 
liaisons and school administrators were invited, but not required to participate. 


Classroom model 


The evaluators worked closely with Cambium’s research department to ensure that the site visit 
rubric reflected PRJ’s conceptual framework and format, and included all the essential 
components of the intervention. The rubric included four components. The first component — 
Section A: classroom environment — provided a descriptive overview of classroom size, desk 
arrangements, technology elements, and materials required for the intervention. These 
elements were not part of Cambium’s IFI, but reflected the evaluators’ experience with the role 
of context in the implementation of education programs. The three other sections were aligned 
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with the IFI. Section B and Section C focused on the quality and amount of instruction and use 
of differentiation strategies, which were the focus of IFI. Section B provided an overview of 
the lesson, while Section C was lesson-specific, and the template changed according to the 
lesson number within the Expedition. For instance, Section C for lessons two, four, six and 
eight included the Word Study element that was not present in lessons one, three, seven and 
nine. Section D had elements of the classroom management component of the IFI. Since the 
observers would not be present during assessment time, all information from the assessments 
was obtained through VPORT. Table 4 displays the alignment between Cambium’s IFI and 
the rubric developed for the site visits (see Appendix B for a copy of the rubric). 


Table 4. Alignment between IFI and evaluators’ observation rubric 


Voyager Index of Fidelity of 
Implementation (IFT) 


RMC Classroom Observation Rubric 


A. Classroom environment 


Quality of instruction 
Amount of instruction 


B. Lesson planning and delivery — Overview 
C. Lesson planning and delivery - Lesson-specific 


Differentiation 


Classroom management D. Classroom behavior/management 
Use of assessments Obtained through VPORT 


During each school visit, two evaluators observed the interventionist during an entire class 
period and each observer completed the observation rubric independently. Following each 
observation, the evaluation team met to discuss their ratings. After the first round of 
observations (approximately 10 classroom observations), the evaluators met to discuss any 
needed changes to the observation rubric and to address questions regarding the observation 
process and ratings. The discrepancies in observer ratings were minimal and did not indicate 
any problems with the overall scoring process. 


The process to calculate the fidelity score for the classroom model followed three steps: (1) all 
observation rubrics completed by every member of the evaluation team were entered into the 
observation database; (2) the different observations for each evaluation team were combined to 
get an average score across all observers for each lesson; and (3) the scores for both rounds of 
observations were then combined to get an average score for each interventionist. Because 
each school had only one teacher, the teacher score equals the school score. Given the limited 
number of observations per school, grade level was not considered in the calculation of scores. 
Based on feedback from Cambium, the evaluators established the fidelity levels in the 
following manner: scores below 0.70 were defined as inadequate or low implementation, 
scores between 0.70 and 0.89 reflected medium fidelity; scores of 0.90 or above were 
considered high fidelity. 


As data from the site visits started to be collected, the evaluators observed that classroom space 
and instruction time (both length of daily instruction and days dedicated to instruction) differed 
broadly from one school to another and strongly influenced the pacing of instruction. Two 
hypotheses were discussed on how to incorporate this finding into the fidelity of 
implementation model. The first hypothesis considered context as a moderator of the 
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classroom implementation. That is, context would favor or hinder how the classroom 
instruction model was implemented, but had no impact of its own. The second hypothesis 
considered fidelity of implementation as a function of the professional development received, 
the implemented classroom instruction, and school context, including classroom size, time 
dedicated to instruction, and days dedicated to instruction. This hypothesis is reflected in 
Figure 4. 


Figure 4. Fidelity of implementation framework 


Instruction 


Context 


Plans were made to collect further data on the role of context in the implementation as the 
evaluators returned to the schools for two more visits before the end of the implementation 
year. Plans were also made to collect attendance data that would provide a more reliable 
source of information regarding the intensity of intervention for individual students. However, 
with the abrupt cancellation of the grant, priorities were placed on maintaining the goodwill of 
the school administrators and interventionists and conserving funds for data analysis and 
reporting. Plans for further data collection on implementation were cancelled, and the two 
final site visits were also cancelled. The final model for calculating the index of fidelity of 
classroom implementation, displayed in Table 5, considered context as part of the overall 
classroom implementation model, and is limited to what was observed during the site visits. 


Table 5. Calculating the classroom implementation fidelity score 


: Total Possible 
Section Score Weighted Score 


Section A .20 X,/6 
Section B 30 Xp/(12 — number of N/A) 
Section C 30 Xc/8 
Section D 20 Xp/(total time intervals — number of N/A) 
Total possible score 
Levels: 0.0 —0.69 = low 0.70 — 0.89 medium 0.90 — 1.0 = high 
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Implementation Year 1 (2010-2011) 
Control students’ instruction during intervention period 


According to the MOA the schools signed with VDOE, all control group students were 
enrolled in elective classes that did not contain supplemental reading instruction. The 
evaluators checked student enrollment in the treatment group in January and March 2011. 
Except for one school, where two control group students were receiving treatment, all other 
control group students were attending elective classes that were not related to supplemental 
reading instruction. A decision was made to maintain these students in the impact study. 


Other Tier 2 interventions for study students (treatment and control) 


As part of their regular instructional practices, the participating schools provided a number of 
remediation or recovery supports to students who did not attain proficiency in the statewide 
assessments. These supports included: (a) remediation during the students’ regularly 
scheduled English or mathematics classes; (b) tutoring by classroom teachers before or after 
school, during Saturday school, and/or during summer school; and/or (c) an additional class 
period during the school day (not for students participating in the Striving Readers study). 
These interventions were geared toward the statewide assessments, rather than improving the 
reading skills of struggling readers. Furthermore, none of the interventions fell under the 
definition of reading programs, nor were they provided by reading specialists. 


Context of PRJ implementation 
Interventionists 


Nine interventionists were hired by the VSRII project, one per school, following the 
procedures established at the outset (see Part J). All teachers had a valid Virginia teaching 
license with teaching endorsement that included grades seven and eight. The interventionists 
taught from two to six classes daily divided between Journeys I (for seventh grade students) 
and Journeys IT (for eighth grade students). 


One interventionist left in March 2011 for a more stable (not grant-funded) teaching position. 
The potential cancellation of the study was already being discussed, and for this reason, the 
school division did not hire a new interventionist. The position was provisionally filled with 
two substitute teachers who had previous training in PRJ. The other interventionists remained 
with their classrooms until the end of the school year, although most of them were actively 
looking for jobs as soon as the news that funding for the study had not been renewed was 
communicated to all VSRII participants at the end of April 2011. Seven of the remaining 
interventionists were re-hired for other teaching positions within the same school division and 
at least one continued providing PRJ instruction. Of the nine VSRII schools, five are using 
general funds to continue PRJ in SY 2011-2012, with three using Journeys I and IT, and two 
using Journeys Beginnings (sixth grade). 
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Classroom space 


A total of 913 seventh and eighth graders were eligible for the study; 457 students were 
randomly assigned to the intervention group and 456 to the control group. Details about the 
random assignment process are provided in Part II. Table 6 presents the student enrollment 
information for the different schools and grade levels for the end of the school year according 
to VPORT. As the table suggests, approximately 12 percent of the treatment group left the 
program with time. This information reflects the number of students that were actually entered 
into the system as participants in the intervention and remained in the list (input into VPORT 
was interventionists’ responsibility), and not the number of students who took the outcome 
assessments. As the table indicates, each interventionist taught from two to six classes with 
average enrollment per class of 9.2 to 21.0 students. The average size of VSRII classes was 
13.2, below the 15 to 20 students suggested by Cambium. 


Table 6. Treatment class size per school and grade 


Average 
taught Grade 7 Grade 8 Total students/class 

1 5 9.2 
2 6 10.5 
3 4 10.0 
4 4 12.5 
5 4 11.3 
6 2 17.5 
7 2 21.0 
8 4 13.8 
9 2 10 

13.2 


Source: VPORT enrollment records 


All teachers had “permanent” classrooms, that is, they had regularly scheduled space to 
conduct the intervention. However, not all classrooms were deemed adequate for the 
intervention by the interventionists and evaluators alike. One school assigned a computer lab 
where the computers occupied most of the desk, thus blocking the students’ view of the 
interventionist and leaving almost no space for the students to complete work. This was the 
school that lost the interventionist by March 2011. In another school, the interventionist was 
teaching in a windowless room with inadequate space for the teacher to move and for small 
group instruction. Two other schools also assigned classrooms that were not adequate for 
organization of small group spaces. Therefore, of the nine PRJ classrooms, four were deemed 
by the evaluators as non- conducive to instruction in general, and particularly to an intervention 
that strongly relied on small group work. 


32 


Virginia Striving Readers Intervention Initiative, Year One Implementation 


Instruction time (Dosage) 


Three factors were found to influence instruction time: minutes dedicated to instruction each 
day (or allotted classroom time), days dedicated to instruction within the school year, and 
classroom attendance. 


Allocated instruction time: Table 7 displays the allotted daily classroom time dedicated to PRJ 
instruction, that is, the official minutes dedicated to intervention. As stated before, 50 minutes 
was the recommended daily instruction time, with an adaptation for 90 minutes provided by the 
developer. Most classrooms offered the minutes indicated by the developer, except for school 
4, where the daily time dedicated to the intervention was 10 minutes shorter than the required 
time. 


Table 7. Allocated time for instruction 


aE? ; Schools 
Time in minutes 1 5 


Classroom time — 
Grade 7 


Classroom time — 
Grade 8 
Recommended class 
time 


Actual instruction days: PRJ is designed to be delivered within one school year, which, in 
Virginia, extends for about 180 school days. However, a number of incidents influenced the 
actual number of instruction days. During the monthly check-ins, the evaluators asked 
interventionists to report the number of days the school was closed as well as the number of 
classes that were cancelled during the time period covered by the check-in. As displayed in 
Table 8, between 4 to 12 classroom days were missed in the Journeys I classrooms, and from 4 
to 13 days in Journeys II classrooms. 


Table 8. School closures and class cancellations 


Grade | Number of Days A Schools 


School closures 


Cancelled classes* 


Missed classes 


School closures 


Cancelled classes* 


Missed classes 


*Schools were open but the class was cancelled for assemblies, test preparation and other reasons. 
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Overall, based on the interventionists’ responses, it does not appear that school closures were 
out of the ordinary for a typical school year. Most of the closures were due to parent teacher 
conferences or adverse weather. Class cancellations were mostly due to test preparation. 
These are activities conducted by the regular content area teachers for English and 
mathematics, or Title I resource teachers, when available, and geared toward all students who 
are at risk of not achieving proficiency in the statewide assessments. The extra lessons are 
used to re-teach standards where the students show the greatest difficulties. 


Student attendance was the final element considered to influence the dosage of the 
intervention. The evaluators had initially planned to request student attendance records from 
the participating schools, but once the Striving Readers grant funding was cancelled the 
evaluators chose not to place this additional burden on the schools and to focus resources on 
obtaining the student achievement data. Nevertheless, during the monthly check-ins, the 
evaluators asked interventionists to report on student attendance. As seen in Table 9, most 
interventionists reported that “almost all of their students attended” classes during the month 
for seventh and eighth grades. School 5 had the most sporadic attendance (33 percent of 
check-ins for both grades). Sporadic attendance was mostly due to student suspension. 


Table 9. Percentage stating ‘“‘almost all students attended class” during the monthly 
check-ins 


Schools 


Implementation of professional development model 
Group training 


Table 10, on the next page, displays the list of professional development conducted during the 
first implementation year, and the number of attendees. The newly-hired interventionists 
attended 16 hours of PRJ launch training on August 9 and 10, 2011, at the Lewis Ginter 
Botanical Garden, Richmond. The training was provided by the Cambium’s Vice-President of 
Implementation Services in the Southeastern Region, with the support of the three VIS 
assigned to the project. Cambium’s Senior Research Analyst was present as an observer to 
ensure that the training was provided with fidelity. The materials that the interventionists and 
the students were to use during the intervention year had already been distributed to the schools 
before the training. The materials included teacher guides, student workbook and library. The 
interventionists were asked to bring the materials and a computer in order to model the lessons. 
To supplement the materials brought with them, each attendee was given a PRJ training 
manual that provided detailed information on the curriculum, the VPORT, the assessments, and 
the implementation process along with appendices of sample schedules and tip sheets. 
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Session 


Table 10. Professional development activities 


8/9/2010 — 
8/10/2010 


16 hours 


Framework and 
components of the 
intervention; 
assessments and 
classroom 
management 


Attendance 


Interventionist 


Other 
LEA liaisons, 
school 
administrators, 
VDOE project 
director, RMC 
evaluation team 
(N=14) 


VSRII Start- 
up 


8/11/2010 


4 hours 


Roles and 
responsibilities, 
management timelines, 
assessment timeline, 
evaluation activities 


As above + 
Cambium 
representatives 
(N=21) 


Launch 
Training 
make up 
session 


8/23/2010 


8 hours 


(interventionist could 
not attend launch 
training) 


GRADE 


8/31/2010 


2 hours 


GRADE 


9/1/2010 


2 hours 


Overview of GRADE, 
how to administer, 
how to score 


4 
(Norfolk & 
Richmond) 


5 
(Roanoke) 


School division 
liaison 


Voyager U 


1/2011 - 
5/2011 


15 hours 


Adolescent 
Vocabulary Strategies 


9 


Voyager U 


1/2011 - 
5/2011 


15 hours 


Adolescent 
Comprehension 
Strategies 


VSRII 
Statewide 
Benchmark 
data meeting 


2/11/2011 


5 hours 


Analyze data from 
Voyager benchmark 2, 
determine the needs of 
schools, teachers, 
students based on the 
data, plan for 
continued Voyager 
implementation 


School 
Principals, 
LEA liaison, 
VIS, 

evaluation team 


Online 
modules 
(optional) 


n/a 


16 hours 
maximum 


Classroom instruction, 
classroom 
management, 
assessments 


Statewide 
meeting 


Summer 
2011 


8 hours 


The meeting was cancelled as the Striving Readers program 
lost fund and the intervention was ending. 


The launch training objectives were centered on three main areas: curriculum, assessment, and 
implementation. The curriculum objectives included: identify each curriculum component and 
describe its intended purpose, prepare to teach a typical 10-lesson Expedition unit, and learn 
how to navigate and utilize the SOLO online component. The assessment objectives were 
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describe each part of the assessment system and its purpose and timeline, administer and score 
each of the assessments, utilize assessments to inform and adjust instruction, and use the 
VPORT online data management system to enter scores and review reports. Finally, the 
implementation objectives included explore the responsibilities of personnel involved in 
implementation, examine groups for whole-group and flexible small-group instruction, and 
explore options for arranging and managing the classroom. By the end of launch training, all 
objectives had been met. 


The launch training was followed by a 4-hour session that had two purposes: introduce the 
VSRII participants and clarify the requirements for the implementation and impact studies. 
Interventionists also were required to attend 30 hours of online PD on research on adolescent 
literacy, what was accomplished between January and May 2011. Depending on their needs, 
the interventionists had access to another 16 hours of online modules that reinforced or 
clarified the topics covered during the launch training, such as classroom management, use of 
the benchmark assessments, differentiating instruction, and others. The VPORT data indicated 
that the interventionists used from 5 to 9 of those 16 hours. 


On February 11, 2011, the interventionists attended a five-hour data meeting to discuss 
findings from the formative assessment and how to use these findings to diversify instruction. 
The meeting, held in a conference room at the University of Virginia in Charlottesville, was 
conducted by the Vice-President of Implementation and the three VIS. The presenters 
explained how the data were analyzed and used to improve instruction. After the general 
presentation, interventionists, school principals and school division liaisons got together with 
the VIS to analyze the data from their classrooms/schools and draw conclusions about 
instruction. The meeting planned for the end of the school year was cancelled when USED 
announced that it was terminating the Striving Readers program. 


Table 11 displays the hours of professional development for each interventionist. All 
interventionists completed the launch training. One of the interventionists was unable to attend 
the August launch training, but received instruction later. Since instruction was provided 
individually, the time required for the launch training was reduced from 16 to 8 hours. All the 
interventionists attended the VSRII start-up meeting, and the training on GRADE. These hours 
were extra to the professional development model, as they were not connected to PRJ 
implementation, and were integrated into “other VSRII PD” hours in the table below. 
Additionally, the interventionists completed different hours of the optional online PD. 


Table 11. Hours of professional development 
Schools 


Launch training 


Adolescent literacy 
Other VSRIT PD 
Online PD (optional) 


“The interventionist attended an 8-hour individual makeup session; the index was calculated on required 43 hours. 
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The VSRII Project Director, LEA liaisons and school principals or representatives were invited 
to attend the launch training, the startup training, the data training, and the end of the year 
training. Two principals attended the launch training, but one had to leave approximately 
halfway through to attend to school-related demands. All schools but one were represented by 
the principal or an assistant principal in the meeting held at the end of the launch training to 
discuss the proposed evaluation activities. The LEA liaisons were present at all activities. 
Considering the focused nature of the intervention, participation in professional development 
for other than the interventionists was not considered part of the fidelity of professional 
development model. 


Individual Supports 


Voyager Implementation Specialists (VIS) were contracted to provide a maximum of 50 hours 
of on-site coaching to each interventionist (10 visits per teacher with 5 hours per visit). All 
interventionists reported receiving regular visits from the VIS. The evaluators talked with all 
three VIS and the VSRII Coordinator regarding the individual supports that had been offered to 
the interventionists. Interviews with VIS and the interventionist check-ins provided conflicting 
information regarding hours offered. Despite attempts to clarify the information, conflicts 
could not be resolved. 


From the interventionists’ perspective, weather conditions that led to school cancellations were 
the principal reason cited for not receiving coaching visits during the period covered by the 
check-in. From the VIS’ perspectives, the main reason for missing coaching days was the 
difficulty to coordinate time with the schools or the interventionists for the coaching visit. One 
school in particular was reported to be blocking the VIS from conducting coaching visits, 
unless a central office staff was present during the visits. Additionally, all interventionists 
reported receiving coaching from the VIS through email and phone. Although these visits 
addressed the needs of each interventionist, interviews and check-ins indicated that three topics 
dominated the conversation: analyzing student data, modeling instruction, and classroom 
management. 


Table 12. Hours of coaching from Cambium Learning Group 
Schools 


Months reported 


% of monthly visits 


Number of expected 
hours VIS coaching 


Number of actual 
hours VIS coaching 


Ratio actual/expected 0.81 


Table 12 includes information related to the coaching that each interventionist reported 
receiving from the VIS. To estimate the intensity of supports, the evaluators calculated the 
total number of coaching hours that the interventionists reported receiving from the VIS 
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relative to the total number of hours that would be expected according to the total number of 
check-ins. For example, School 1 had six check-ins, so it would be expected that the 
interventionist should have received 30 hours of coaching (5 hours per check-in). The 
interventionist reported receiving 37 hours of coaching, in excess to the expected 30 hours. 
Part of this discrepancy can be attributed to the fact that interventionists received coaching 
support via e-mails and conference calls, two activities that were not included in the 
calculation. 


Until leaving the position in April 2011, the VSRII Coordinator conducted monthly visits with 
interventionists and school division liaisons. Each interventionist received an average 12 hours 
of individualized support between October 2010 and March 2011, when the last visit occurred. 
During the visits, the VSRII Coordinator observed the interventionists teaching a lesson and 
provided feedback. Topics also included classroom management, assessment, use of VPORT, 
and planning. These hours were in addition to the professional development included in VSRII 
logic model. 


Index of fidelity of implementation: professional development model 


Table 13 summarizes information from the professional development and supports and 
includes the index for fidelity of implementation for the professional development model. As 
the table indicates, during the first year of the VSRII implementation, each interventionist 
received between 72 and 127 hours of professional development that included face-to-face and 
online training, and individual supports from VIS and VSRII Coordinator. Considering the 
discrepancies in the data collected and the many factors that influenced the presence or absence 
of the individual supports in a specific school, coaching hours were not incorporated into the 
final professional development index. 


Table 13. Index of fidelity of implementation — professional development model 
Schools 
5 


Hours of individual supports 
(VIS)! 

Hours of individual supports 
(VSRII Coordinator) * 
Hours of group professional 
development (PD) 


59.0 | 60.0} 57.0} 27.0] 57.0) 49.0 | 58.0] 66.5) 59.5 
51.0} 51.0) 51.0} 51.0} 51.0) 43.0} 51.0} 51.0) 51.0 
asc 


*Not included in the index calculation 
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Based on the required professional development hours, the index of fidelity of professional 
development ranged from 0.5 (school 4) to 1.3 (school 8). All interventionists, except for 
School 4, received an index at or above 1.0, which was defined as adequate. It is unclear 
whether School 4 interventionist truly did not attend all of the required hours of PD, since that 
interventionist left the project in March 2011. It is possible that the interventionist did 
complete the online adolescent literacy but did not have an opportunity to report it to the 
evaluators before leaving the position (the evaluators had no access to the VoyagerU 
information). 


Implementation of the classroom model 


As previously discussed, PRJ J and II are formatted as a series of lessons, or Expeditions, each 
comprised of ten 50-minute lessons. The initial expectation was that interventionists would 
cover one lesson per day, such that they would complete one Expedition every ten days, with 
an optional day for re-teaching. Interventionists could decide to use this extra day of 
instruction based on student need, based on results from the formative assessments. For 
classrooms with 90 minute instruction, the interventionists were expected to cover two lessons 
a day. In all cases, Cambium staff provided suggestions on how to modify the lessons to fit the 
classroom period. 


At the training, the overall understanding was that the interventionists should complete one 
lesson per class period without modifications. Pacing became a major challenge for many, if 
not all interventionists. As a result, the VIS suggested either completing the remainder of the 
lesson at the next class period or making modifications as appropriate to cover the entire lesson 
in a day, while still keeping with the scaffold structure and providing as much reading time as 
possible to students. Yet, during the check-ins with the evaluators, the interventionists 
expressed confusion about whether they should include adaptations to finish the lesson in one 
day or make no modifications and carry over any activities to the following day as needed. 
They also felt that they were receiving conflicting advices from VIS and VSRII Coordinator. 


The evaluators conducted two visits to each of the VSRI/ schools to monitor classroom 
instruction—one in October 2010 and the other in February 2011—for a total of 47 
observations, with five or more lessons observed for all but one interventionist. Due to 
scheduling conflicts, the evaluators were unable to observe the interventionist at School 3 
during the first round of visits, even after multiple attempts. Table 14 lists the number of 
lessons observed, time available for PRJ instruction, and number of lessons completed during 
the observation (the evaluators stayed for the full class period). 


Table 14. Lessons completed within class time 
Schools 


Number of lessons observed 


Average class length (minutes) 


Expected class time (minutes) 
Lesson Completed in the period* 
*Number of lessons completed across all observations 
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Almost all interventionists had sufficient scheduled class time to complete a full lesson, with 
the exception of School 4, which had only 40 minutes. However, as discussed before, 
allocated class time did not necessarily correspond to actual instruction time. Those 
interventionists who taught PRJ during the first period had their actual class time reduced 
because of morning announcements, while interventionists with afternoon classes often saw 
their class periods cut short due to early release schedules, assemblies or other activities. In 
fact, during the check-ins with interventionists the issue of pacing was one of the most 
commonly-cited problems. This was echoed by the three VIS and the VSRII Coordinator. 


Two of the VIS noted that student off-task behavior was a significant hindrance to instruction 
in some of the schools and the evaluators had occasion to observe the effect of student 
behavior on instruction. For instance, in one of the classroom visits in October, the evaluators 
noted that due to student disruptions or other off-task behavior, the interventionist in School 9 
was unable to complete even one lesson, despite having a longer period for instruction. The 
last row in Table 14, above, provides some indication of the impact of behavior and actual 
classroom time on completion of lessons. As can be seen, only one interventionist was able to 
complete the lessons during all the observations. 


The three VIS worked with the teachers to revise the pacing calendars for each of their classes, 
as necessary, but pacing was a challenge throughout the intervention. The VIS expected that 
the interventionists would improve pacing as they became more familiar with the intervention. 
Therefore, the expectations were that pacing would become less of a problem in the second 
implementation year. 


Actual student use of technology 


The VSRII budget included money for the purchase of software and hardware. During SOLO, 
an activity that occurred in Lessons 5 and 10 of each Expedition, the students had the 
opportunity to practice on the computer the skills they had learned during the week. The 
evaluators had the opportunity to observe two SOLO lessons (one in grade 7 and one in grade 
8) and noted that students were engaged in the task. However, in some of the check-ins, the 
interventionists remarked that students got “bored” with the SOLO lessons, especially when 
they had to repeat sections they had already completed due to problems with the software. 
Additionally, interventionists reported having problems with the online connection. As one 
interventionist observed, “SOLO has become a very big issue for my students; it continues to 
freeze and lock my students out.” To avoid problems with the time it took students to load the 
program, interventionists started asking students to log-in to the SOLO at the beginning of class 
so that it would be ready to use when needed. 


Interventionists also reported having difficulty with the DVDs provided for the intervention. 
DVD segments were included in Lessons 1, 6 and 9 of each Expedition, and covered topics 
designed to engage students into further reading. The evaluators observed interventionists 
struggling to find the correct location on the DVD to correspond with a particular lesson or 
having problem with the DVD not loading correctly. Cambium was responsive to 
interventionists’ requests for assistance with the DVDs, and it is the evaluators’ understanding 
that Cambium even made some changes to the DVDs based on the interventionists’ feedback. 
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Further, at the request of the interventionists, Cambium prepared electronic copies of the 
student worksheets and other handouts so that the interventionists could display them on 
interactive white boards. 


Actual intensity of intervention students received 


Table 15 displays the number of Expeditions completed during the school year, as reported in 
VPORT, and the ratio between the number of Expeditions completed and the total number of 
Expeditions in the intervention. The required number of Expeditions and the number of 
Expeditions completed were the same across the two grade levels and therefore, only one grade 
is presented. Additionally, it is important to observe that information on VPORT was entered 
by the interventionists and reflect mostly when the assessments were conducted, rather than the 
final day of class. As seen in the table, the most an interventionist was able to cover was 80 
percent of the Expeditions, while the least amount covered was 30 percent. 


Table 15. Number of Expeditions completed 


* 
Expeditions j Schools 


Completed 


Maximum 


Ratio 


*Allotted time of instruction: Schools 6-9 = 80 to 90 minutes; Schools 1, 2, 3, 5 =50 minutes; School 4 = 40 minutes. 


The actual intensity of the intervention the students received appeared to have been affected by 
four primary elements: (a) actual instruction time; (b) the interventionists’ skills in maintaining 
pacing; (c) actual days of instruction, and (d) student attendance (or the need to re-teach for 
absent students). Interestingly, despite having a shorter class period (40 minutes, with 
approximately 35 minutes of actual instruction time), the interventionist at School 4 was able 
to cover almost three-fourths of the Expeditions. The VIS observed that the School 4 
interventionist was doing “an exceptional job moving the lessons forward.” The observations 
suggested that the interventionist was indeed skillful with maintaining pacing, but not enough 
observations were made (five lessons throughout the year) to confirm that statement. 


Another factor that merited further exploration was the impact of block scheduling on the 
interventionists’ ability to maintain pacing. As Table 15 suggests, the interventionists from 
schools with block scheduling (schools 6-9) completed fewer Expeditions than their colleagues 
from schools where class time ranged from 40 to about 52 minutes. Unfortunately, the early 
cancellation of the grant impaired the evaluators’ ability to further investigate the relationship 
between block scheduling and slower pacing. 


Student assessment procedures actually carried out 
As detailed in Part J, the PRJ assessment system includes measures of text fluency and 


comprehension, end-of-lesson assessments, progress monitoring, and student self-assessments. 
The assessments were conducted as planned. The Reading Benchmark I was conducted at the 
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beginning of the school year and was used to place students in the appropriate level of reading 
materials and in one of three appropriate levels of text in SOLO. Reading Benchmarks II and 
III were administered in January and May. Comprehension and vocabulary assessments were 
administered at the end of each Expedition and the scores were recorded in VPORT. In the 
monthly check-ins, interventionists reported using student assessment data to choose 
appropriate re-teaching opportunities targeted to specific skills where students demonstrated 
difficulty. The re-teaching activities were conducted on day 11 of the Expeditions. The school 
divisions also used the GMRT and SOL assessments for grades 6 through 8 to determine 
eligibility to the study and for the impact study, as discussed in Part III. The GMRT was 
administered in April 2011, while the SOL English/Reading was administered in May 2011. 


Teacher use of ongoing assessments 


The VIS and the interventionists confirmed that they were using the PRJ-specific assessments 
to guide instruction. Assessment results were posted on VPORT. The evaluators avoided 
coming during assessment days and they were also unable to observe planning time. Table 16 
summarizes one of the data reports provided by VPORT. 


Table 16. Journeys I and II — benchmark data 
Seventh Grade Scores Eighth Grade Scores 


B2 Avg. Expedition Bl B2 Avg. Expedition 
Lexile | Gain Complete Lexile | Lexile | Gain Complete 


1 
2, 
3 
4 
5 
6 
7 
8 
9 


In February 2011 Cambium hosted a data meeting for the interventionists, during which the 
trainers talked about the assessment data and provided instruction on how to interpret the data 
and use the results to inform instruction. In the monthly check-ins, interventionists mentioned 
that the data meeting was helpful and informative and that they liked getting the opportunity to 
compare their student data with that of the other interventionists. 


Index of fidelity of implementation for the classroom model 


Table 17, on the next page, displays the average scores obtained for each item of the classroom 
observation rubric. Sections A (context) and B (general lesson plan and delivery) were scored 
on a two-point scale whereby 0 = inadequate and 1 = adequate. Section C (specific lesson plan 
and delivery) used a three-point scale for C1 and C2, and a two-point scale for C3. Section D 


42 
Virginia Striving Readers Intervention Initiative, Year One Implementation 


(classroom behavior and management) reflected percent of time where the specific behaviors 
were observed. 


Table 17. Scoring for the classroom observation rubric 


Schools 
1 D 3 4 5 6 7 8 9 
Number of classes observed 5 5 2 5 5 8 6 6 5 
Section A: Context 
Al - Sufficient space : 1.20 | 1.00 | 1.00 
A2 - Instructional areas : 1.20 | 1.00 | 1.40 
A3 - Teacher resources . 2.00 | 2.00 | 2.00 
A4 - Student materials : 2.00 | 2.00 | 1.60 
Section score: X,/8 : 0.80 | 0.75 | 0.75 
lesson plan and delivery 
B1 - Follows curriculum guide : 2.00 | 1.70 | 2.00 
B2 - Brisk pace : 2.00 | 2.00 | 2.00 
B3 - Skills modeled : 2.00 | 2.00 | 2.00 
B4 - Correction procedures : n/a 1.50 | 1.00 
BS - Students in groups : 1.00 | 2.00 | 1.60 
B6 - Differentiation 2.00 | 2.00 | 2.00 
Section score: Xp/(12 —N/A) . 0.90 | 0.93 | 0.88 


C1 — Components delivered in 
order 
C2 — Steps delivered in order , 2.57 | 2.75 | 2.11 | 3.00 


2.75 | 3.00 3.00 | 3.00 


score ane 0.60 | 1.50 | 1.24 | 1.63 
suggested timeframe 


Section score: Xc/8 ; 0.74 | 0.91 0.79 | 0.95 
Section D: Classroom behavior and management 


0.60 | 0.77 | 1.00 


D1 - % time students pay 
attention 

D2 - % time students respond 
to prompts 

D3 - % time students actively 
participate 

D6 - % time students follow 
expectations for group work 
D4 - % time teacher addresses 
behavior (x2) : : ; : 2.00 
D5 - % time teacher involved 
engaging students (x2) : 2.00 
Section score: Xp/(total time 0.99 
intervals — number of N/A) : 


0.58 | 0.72 | 0.90 


0.60 | 0.67 | 1.00 


0.55 | 0.70 | 1.00 
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Each section received a weight that addressed the importance of the section within the overall 
classroom implementation model, as adapted from Cambium’s IFI. Table 18 displays the 
results. As the table shows, based solely on the adequacy of delivery observed during the site 
visits, two of the nine interventionists were classified as attaining high fidelity of 
implementation (score above 0.9), while the remaining seven attained adequate (medium) 
fidelity of implementation levels. 


Table 18. Fidelity to the classroom instruction model (weighted) 


: Schools 
Sections 


Classroom Fidelity 0.86 
Index 


Fidelity of implementation scores 


Table 19 displays the scores for the fidelity of implementation of the professional development 
and classroom models. According to these indices, all but one school attained adequate 
implementation of the professional development model. Regarding the classroom model, 
seven schools attained an adequate (medium) implementation level, while two attained high 
fidelity. 


Table 19. Fidelity scores per school 


hool 


PD . 1.12 
Instruction 0.81 
Levels 


PD 
Instruction 
Legend: I= Inadequate; A = Adequate; H= High 


Factors Influencing Fidelity of Implementation 


Two groups of factors were found to influence the implementation process: general factors that 
either facilitated or hindered the fidelity of implementation across all participant schools, and 
factors that were specific to one or a small group of participant schools. 


General factors 
Among the three factors that influenced the project implementation in general, planning year 
and professional development and supports were seen as facilitators. Alternatively, the abrupt 
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ending of the Striving Readers grant created a challenge for the schools, as it will be explained 
below. 


Planning year: USED incorporated one full planning year under the Striving Readers funding. 
From the perspective of the implementation, the planning year provided time for the 
representatives from the state, LEAs and schools to talk among themselves and with the 
developers, ask questions and share concerns. The developers had time to address the 
questions brought by the school personnel before the actual instruction started, and propose 
adaptations to the intervention to address the uniqueness of each school. At the start of the 
implementation year, all participants had a clear idea of their roles, responsibilities and 
expectations. They also had become familiar with the intervention. From the perspective of 
the study, the planning year was essential for the evaluators to build trust with the 
implementers, learn about the different policies and regulations that could affect the study in 
each of the participant school districts, address questions and assuage fears from school 
administrators and school division representatives. Furthermore, the planning year provided 
the evaluators with enough time to become familiar with the intervention and to develop and 
test data collection instruments that were faithful to the intervention model. 


Intensive coaching and supports: As previously discussed, the VSRII project included 
approximately 50 hours of on-site coaching services for each interventionist, or approximately 
one visit per month. Coaching activities included modeling lessons, observing lessons and 
providing feedback, discussing student data and how to use the data for lesson planning, 
clarifying questions, and supporting the interventionists as they implemented the intervention. 
During the interviews, the VIS mentioned that they could see improvements in the 
interventionists’ classroom instruction following the coaching visits. As one VIS explained, 
the coaching provided opportunity to “really see how the teachers are teaching and actually 
coach them based on observations of whole lessons.” As part of the overall supports, VDOE 
dedicated a full time staff for the project and the two people who held the position at different 
times demonstrated a high level of dedication. Monthly phone calls and face-to-face meetings 
started from the planning year. These activities were essential to bring all participants together 
and highlight VDOE’s commitment to the success of the project. The second VSRII 
Coordinator also had a role in offering individualized supports to the interventionists that 
complemented the VIS supports. 


Elimination of the Striving Readers Grant Funding: With the announcement in April 2011 that 
funds for the grant had been cancelled, the VSRII Coordinator, a position fully paid by the 
grant, decided to find a more stable position and left the project one month later. Since the 
grant was coming to an end, VDOE decided to eliminate the position, and the Grant Manager 
assumed the sole leadership for the project. At the schools, the school administrators started to 
make provisions for other initiatives that would replace PRJ, or search for new source of funds 
to continue with the intervention. As for the interventionists, knowing that the position would 
no longer be available, they started looking for jobs. Two comments from interventionists 
summed up the general feeling among VSRII participants when they heard the news of the 
cancellation: 
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In the end, the students who hung in there with me and worked consistently got 
the greatest increase in Lexile scores. I have really grown to love my readers 
... and Iam concerned that there will be no further reading safety net for those 
who made progress but are still not at grade level. 


I have seen the growth of my students this year and hate to see the end of this 
program. 


Specific factors 

Reviewing Table 18 (p. 44), two components of the classroom model showed the greatest 
variance: context (Section A) and classroom management (Section D). These two components 
appeared to behave not as factors in themselves, but as moderators for lesson plan and delivery 
(Sections B and C) by creating a “disturbance” in the model. That is, although interventionists 
were implementing the intervention in relatively similar ways, context and classroom 
management might explain why they ended up completing different numbers of Expeditions 
with somewhat different fidelity scores. These factors are discussed below. 


Context: Under context, actual time of instruction was the component that appeared to bring 
the greatest variation to the classroom implementation model. Actual time of instruction is 
defined as allocated time for instruction (or classroom time) minus time taken away from 
instruction due to interruptions. 


Regarding allocated time for instruction, one interventionist had 40 minutes of class time, 10 
minutes fewer than the 50 minutes a day recommended by the developer. Interesting enough, 
this interventionist appeared to have completed 11 Expeditions, a high number compared to 
peers. The unexpected finding was that interventionists with 90 minutes of class time 
completed even fewer Expeditions than their colleagues with 50 minutes. This finding merited 
further investigation in the second implementation year, particularly due to the fact that many 
schools are moving into block schedule and the impact of such schedule on lesson planning 
should be explored. It is of note that even when interventionists had sufficient allocated class 
time for the PRJ lessons, class time was frequently cut short by announcements, assemblies, 
time spent walking students from one class to another, early release schedules, and/or student 
behavior issues. Regarding these interruptions to classroom time, classes scheduled in the first 
and last periods of the day were more prone to being curtailed (announcement s and early 
releases) than those scheduled in the middle of the day. 


Days allocated to instruction were another component under Context that varied across 
schools. Classes were cancelled due to parent-teacher conferences, field trips, testing, test 
preparation, snow days, or assemblies. Preparing students for the statewide assessments was 
probably the most frequent cause of cancellation of PRJ classes. As one interventionist 
commented, “J am hoping that when they get through testing ... that I will see even better 
progress.” 


Technology glitches was the third element considered under Context that elicited numerous 
comments at beginning of the intervention, and negatively influenced time of instruction and 
the teachers’ ability to maintain the required pacing. During the first visit in October, some 
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interventionists still needed to receive the computers that had been ordered using grant 
funding, while others had to coordinate with their local technology support staff to obtain 
access to the online components. The interventionist in School 5, for example, reported not 
having access to the SOLO online passwords in late October 2010, almost two months after the 
start of the intervention. In addition, low bandwidth in some schools delayed the time for 
students to log onto SOLO. Computer “freezes” or other technology glitches that disrupted 
instruction were observed during the site visits and frequently reported during the check-ins. 
Interventionists also had difficulties with the DVDs provided with the PRJ and at times were 
not able to use them in the appropriate part of the lesson. As mentioned above, technology 
glitches per se where not the reason for disrupting instruction. In some classes, interventionists 
found a way to minimize the impact of technology malfunctioning. For instance, in one 
classroom, the students would log on to the computer as soon as they entered class to give time 
for the log on to be completed before the students needed to work on the computer. In other 
classes though, either because of the level of the malfunctioning or the teachers’ ability to deal 
with it, technology glitches were observed to raise the level of frustration of interventionists 
and students alike, and negatively impact the implementation. 


Classroom management: Classroom management is defined as the interventionists’ ability to 
engage student into the learning process. Student behavior was an important factor affecting 
the ability of interventionists to deliver the lessons in the way that was intended by the PR/ 
developer. During the visits the evaluators observed numerous instances of students’ 
disruptive and off-task behavior. The interventionists, VIS, and VSRII Coordinator agreed that 
disruptive behavior had a considerable impact on the implementation of the intervention over 
the course of the school year. When students acted out the interventionists had to address the 
student behavior, thereby interrupting the delivery of the lessons. As one VIS commented, “/in 
some schools] the primary problem is behavior management; the teacher loses between 20-35 
minutes each day to dealing with behavior problems. [It affects] everything from pacing to 
outcomes.” The interventionists cited two main reasons for what seemed unusually disruptive 
behavior. First, they commented about a large number of students with emotional and 
behavioral disorders in their classrooms (six in one case) without access to paraprofessionals. 
Another reason also cited was the structure of the intervention. The interventionists reported 
that students got bored with the repetitive nature of the lessons and their attention waned, often 
resulting in misbehavior. One interventionist commented that, “Students are getting tired of 
the lessons; they know what to expect and are getting tired of the repetitiveness of the 
[Expeditions].” Yet, interviews and observations suggested that “boredom” differently 
influenced the classrooms. While one interventionist saw repetition as a hindrance, another 
saw it as a positive feature of the program, commenting “The students moan and groan about 
the repetition of activities but also seem to fall into activities by habit most of the time.” 


Ramifications of implementation results for impact analyses 


Findings from the first implementation year suggest that the interventionists received the 
planned professional development and supports, with the exception of one school, where the 
interventionist left the school before the end of the school year. Classroom instruction was 
similar across interventionists, from the perspective of delivery of instruction, with two schools 
attaining high fidelity while the remaining seven schools attained adequate or medium fidelity. 
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Interviews and observations found no differences in the implementation of the intervention for 
students in seventh and eighth grade. However, intensity of instruction (dosage) varied across 
schools, with interventionists completing from 30 to 80 percent of the curriculum by the end of 
the school year. It must be noted that none of the assessments used for the impact study were 
conducted at the end of the school year (June 2011). The SOL English/Reading is traditionally 
conducted in mid-May, while GMRT was conducted in April. It is not clear that another 
month of classes would influence assessment results, but studies where the curriculum is 
allowed to be completed before the post-assessment is implemented may further the 
understanding of the impact of reading programs on adolescent struggling readers. 
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PART II: IMPACT STUDY 


Study Design 


The supplemental literacy intervention in the VSRII was a one-year intervention for seventh 
and eighth grade students who were two years below grade level in reading achievement. 
Due to the termination of funding for the Striving Readers program, the intervention was 
provided for only one year. The evaluation of the one-year impact of this intervention is 
based on an experimental design, with random assignment, to address the following research 
question: 


¢ What is the impact of participating in the PRJ supplemental literacy intervention for one 
school year on the achievement of striving readers in grades 7 and 8? 


The intervention’s impact at the end of the 2010-2011 school year was estimated by calculating 
the difference, adjusted for student level covariates, between the average reading achievement 
of eligible students randomly assigned to the intervention (the treatment group) and the rest of 
the eligible students who participated in other activities, not to include any supplemental 
literacy instruction (the control group). The Gates-MacGinitie Reading Tests (GMRT) Ae 
Edition and the Virginia Standards of Learning (SOL) English/Reading assessment where used 
to measure the reading achievement of treatment and control students. 


Sample selection 


As described in the Introduction, nine middle schools serving students in grades 6 through 8 
from three school districts participated in the study. In the spring of 2010, the schools 
administered the GMRT 4" Edition to all students in grades 6 and 7. Students whose test 
results were equivalent to two or more grades below their grade level were eligible for the 
VSRII intervention. Additionally, students in sixth and seventh grades who did not pass the 
2010 SOL English/Reading assessment were also eligible for the intervention regardless of 
their GMRT scores. Two types of students were excluded from the eligible pool before 
random assignment to the treatment and control groups: students with an Individual Education 
Plan (IEP) that precluded their participation in the study, and students whose parents requested 
their children be exempt from the study. 


In the spring of 2010, the total sixth and seventh grade enrollment for these nine schools was 
3,680. Over the summer, the school districts provided lists of students who were at least two 
years below grade level on the GMRT and/or failed to pass the SOL, after the exclusion criteria 
described above were applied. The lists contained 918 students. For each grade within each 
school, these students were randomly assigned to the treatment and control groups. The total 
number of students assigned to each group was 460 and 458, respectively. After random 
assignment, it was discovered that five students (3 treatment and 2 control) were missing 
evidence that they were eligible on either measure. These five students were designated 
ineligible after random assignment and were removed from the Intent to Treat (ITT) sample. 
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Of the remaining 913 students (457 treatment and 456 control), 514 were at least two years 
below grade level on the GMRT although they did not fail the SOL (or were missing baseline 
SOL results), 117 failed to pass the SOL but they were not two years below grade level on the 
GMRT (or were missing baseline GMRT results), and 282 were eligible on both measures. 
These three eligibility groups were relatively well-balanced between the treatment and control 
groups. The percentage of treatment students in each group was 51.2 percent, 44.4 percent, 
and 48.9 percent, respectively. There were 481 students in grade 7 (49.7 percent in the 
treatment group) and 432 in grade 8 (50.5 percent in the treatment group). 


Figure 5. Consort diagram of eligibility, random assignment, and attrition 


POPULATION OF STUDENTS 
N= 3680 


ELIGIBLE STUDENTS INELIGIBLE STUDENTS 
n= 918 n= 2762 


RANDOM a 
TREATMENT GROUP CONTROL GROUP 
| n= 460 n= 458 | 


5. 
Not Eligible for Study Post-RA Eligible for Treatment Eligible for Control Not Eligible for Study Post-RA 
n= 3 n= 457 n= 456 n= 2 


Targeted Sample Targeted Sample 
at Baseline at Baseline 
n= n= 456 


f 


Final Intent-to-Treat Attrition at Follow-up 
Analytic Sample at n= Variable 
Follow-up Attrition due to: 

n= * No GMRT TR Score 
n= * No GMRT Comp Score 
n= * No GMRT Vocab Score 
n= * No SOL Reading Score 


Attrition at Follow-up Final Intent-to-Treat 


n= Variable Analytic Sample at 
Attrition due to: Follow-up 

* No GMRT TR Score n 
* No GMRT Comp Score 
* No GMRT Vocab Score 
* No SOL Reading Score 


a J 2 


As described in Figure 5, of the 913 students in the ITT sample, just fewer than 40 percent did 
not have GMRT scores in Spring 2011 and, thus, were not included in the impact analyses of 
GMRT scores; over 20 percent were missing Spring 2011 SOL scores. The attrition rate for 
treatment students was slightly higher than for controls. Two reasons were presented by school 
personnel for the high percentage of students without a Spring 2011 GMRT scores. First, 
GMRT is not a state mandatory test and students who are absent from school the day when the 
test is administered are not required to take the test another time. Secondly, GMRT was 
administered in April, after the announcement of the end of the grant, and the school 
administrators and interventionists were already moving beyond the study and looking for 
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replacement programs and jobs. It became difficult for the LEA liaisons and evaluators to 
demand make-up days for GMRT. 


With this amount of attrition, it was imperative that an analysis of differential attrition effects 
on these test scores be carried out. A 2x2 factorial ANOVA was conducted to compare the 
Spring 2010 baseline scores on each measure—GMRT Extended Scale Scores for Total 
Reading, Comprehension, and Vocabulary, and SOL Scale Scores for English/Reading—for 
students with and without Spring 2011 scores in the treatment and control groups. In each of 
these four analyses, the interaction between having a Spring 2011 score and experimental 
group was statistically insignificant (p > 0.05), supporting a conclusion that the missing 
students from the treatment and control groups had similar levels of reading achievement at 
baseline. 


Data collection 


Two assessments were used to determine eligibility to the study and to assess outcomes: 
GMRT, 4" Edition, and the SOL English/Reading for students in grades 6 and 7. The 
psychometric qualities of these assessments are discussed in Part I, Assessments (p. 20). 


Study eligibility was assessed with results from the spring of 2010 administration of the 
GMRT and SOL for students in grades 6 and 7. Eligibility was determined by examining Total 
Reading grade equivalents for the GMRT and English/Reading performance levels for the 
SOL. Student reading achievement outcomes were measured with GMRT Extended Scale 
Scores in Total Reading, Comprehension, and Vocabulary and SOL scale scores in 
English/Reading obtained in spring of 2011 for students in grades 7 and 8. 


Student demographic data were also collected from each of the three participating school 
districts with varying success. Gender was available for all students; ethnicity was available 
for students in all schools except one; special education status was available for students in six 
schools; and poverty (free or reduced lunch) and English proficiency (LEP) were available for 
students in five schools. This lack of uniform availability, which reflects different data-privacy 
regulations across the school districts, had implications for the use of this information as 
covariates in the analysis models. 


Data analysis 


Impact analyses of ITT student achievement in reading on the two assessments, GMRT and 
SOL, were conducted to assess the effects of participation in PRJ for the SY 2010-2011. 
These analyses are based on the 913 eligible struggling readers in grades 7 and 8. GMRT 
Expanded Scale Scores for Total Reading, Comprehension, and Vocabulary and SOL scale 
scores for English/Reading were analyzed for the two grades combined and for each grade 
separately. 


Multi-level regression analysis models were used to estimate and test the statistical 
significance of the difference between the reading achievement of students receiving the 
intervention and the control group. Two-level models were employed that express the Spring 
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2011 GMRT and SOL scores as a function of student and school variables.° The student’s 
grade was treated as a student-level covariate in analyses that included students from both 
grades since the GMRT Extended Scale Scores are related to grade level. The treatment 
variable was included at the student level of these models. 


The Spring 2011 GMRT and SOL scores were the dependent variables in these analyses. 
Originally, the 2010 GMRT and SOL scores—representing the same test as the dependent 
variable—were to be included as the principal student-level covariate. Student demographic 
variables were also to be included as covariates in these analyses. However, there were several 
significant missing data problems. First, no Spring 2010 SOL data were obtained for almost 
one-third (299) of the ITT sample. Second, no Spring 2010 GMRT scores for Comprehension 
and Vocabulary were obtained for over half (551) of the ITT sample, as the school division did 
not compute these scores separately. Third, 61 students were missing all scores on the Spring 
2010 GMRT. The number of students missing these scores who also had valid scores for the 
outcome variables was smaller than this (13 or 14 for the 2011 GMRT scores and 39 for the 
2011 SOL scores). Finally, as mentioned above, demographic data for large numbers of 
students were not obtained due to school districts’ restrictions on demographic data. 


The first two types of missing data were addressed by employing the Spring 2010 GMRT Total 
Reading scale score as the student-level covariate for the analyses of 2011 GMRT 
Comprehension and Vocabulary scale scores and the 2011 SOL scale score.’ The missing 
baseline 2010 GMRT Total Reading scores were handled by employing a dummy variable 
adjustment—setting missing cases to a constant and setting a zero-one dummy variable equal 
to one for the missing cases and including the recoded baseline score and the dummy variable 
as covariates in the impact analysis model. 


In addition to the recoded 2010 GMRT Total Reading scale score and the dummy variable that 
flagged missing cases on the recoded scale score, only two other covariates were employed in 
the impact analysis model: A zero-one dummy variable indicating student membership in the 
treatment group and a zero-one dummy variable indicating student grade level.’ Student 
demographic variables were not included due to the large and variable amounts of missing 
data. 


To estimate the impact of the intervention on student reading achievement, the following 
model was tested: 


At the student level, 
Yi = Boj+ BiX wt B2X wt B3X sj + BsX aj t+ ry, where 


Yj is the Spring 2011 test score (GMRT/SOL) for student 7 in school j; 


° Three-level models, employing student, school, and district variables were possible. However, there were only three districts 
and one district had only one school. Consequently, the three-level models were not used. 

’ The correlations between the Spring 2010 GMRT Total Reading scale score and the outcome variables ranged between 0.295 
for the 2011 SOL scale score and 0.406 for the 2011 GMRT Total Reading scale score. 

8 The grade level variable was only employed in analyses based on students from both grades. Scale scores on the GMRT are 
vertically scaled. Thus, some of the variance in GMRT scores could be due to variation in grade level. This variable was 
omitted from the final analysis model if its significance level was greater than 0.20. 
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Xj; 18 an uncentered dummy variable coded 0 for control and | for treatment students in 
school j; 

Xj; is the grand mean centered Spring 2010 “recoded” GMRT Total Reading score for 
student 7 in school j; 

X3;; 1s a grand mean centered dummy variable coded 1 for students missing the Spring 
2010 GMRT Total Reading score and 0 otherwise; 

X4ij is a grand mean centered dummy variable coded 1 for students in g grade and 0 
otherwise; 

{oj 1s the adjusted mean Spring 2011 test score for the control students in school j, 
controlling for the Spring 2010 GMRT Total Reading score and other covariates; 

fis the adjusted difference between treatment and control group mean Spring 2011 
test scores (the PRJ treatment effect, controlling for the Spring 2010 GMRT Total 
Reading score and other covariates); 

fh is the slope of the regression of Spring 2011 test scores on Spring 2010 GMRT Total 
Reading scores; 

f3 is the adjusted difference between the mean Spring 2011 test scores of students 
missing the Spring 2010 GMRT Total Reading score and those not; 

f4 is the adjusted difference between the mean 7" and 8" grade Spring 2011 test scores; 
and 

rj 18 a unique effect for student 7 in school j and is ~ N(0,0”); 


All of the above coefficients at the student level, except fo;, are assumed fixed across schools. 
The adjusted mean Spring 2011 test score for the control students in school j is modeled as a 
function of the student’s school: 


Boi = Yoo + uoj, where 


Yoo 1s the adjusted mean Spring 2011 test score for all control students; and 
Uo; 18 the unique effect of school j and is ~N(0,1). 


The null hypothesis of no PRJ treatment effect on 2011 test scores is Ho: /; = 0 and is tested 
with a f-statistic. 


To estimate the impact of PRJ at the end of year one for students in each grade separately, the 
model was identical to the above except for the omission of £4 X4;, which represents the 
student’s grade. 


Description of the Year One Sample 
Student demographic characteristics 


As mentioned earlier, some of the participating schools did not supply all of the requested 
student demographic data. The data that were received are summarized in Table 20, on the 
next page. There were slightly more eligible students in grade 7, and there were more males 
than one might expect. The important comparison, however, is between the treatment and 
control groups. None of the percentages displayed were significantly (p < 0.05) different 
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between the two groups. Furthermore, numbers of students with missing data did not 
disproportionately represent one group over the other. Also, the missing data were always for 
an entire school. Given that the random assignment was implemented within each school (and 
grade), it is very unlikely that there may have been any differences in the distribution of 
missing data between the treatment and control groups. 


Table 20. Demographic characteristics of the ITT sample 
Student Characteristic Control’ Treatment’ 


Enrolled in Grade 7 (53.1%) (52.3%) (52.7%) 
Enrolled in Grade 8 (46.9%) (47.7%) (47.3%) 


Female (45.4%) (43.5%) (44.5%) 
Male (54.6%) (56.5%) (55.5%) 
Total 
African-American (67.6%) (69.0%) (68.3%) 
Total 
Students with Disabilities (21.1%) (27.4%) (24.3%) 
Total 
Free or Reduced Lunch (89.4%) (86.3%) (87.8%) 
Total 
English Language Learner (7.6%) (8.9%) (8.2%) 
Total 


*Percentages are based on the total number of students with valid data for each characteristic in the control, treatment, or 
total group. 


Table 21. Comparison of treatment and control groups on Spring 2010 
scores on the GMRT and SOL 


Test Score Means Significance 
Control Treatment Level 


GMRT Total Reading Extended Scale Score 


GMRT Comprehension Extended Scale Score 


GMRT Vocabulary Extended Scale Score 


SOL English/Reading Scale Score 


* Numbers in parentheses are the numbers of students in each group having a valid test score. 


Student baseline achievement 


The reading achievement scores of the two design groups were compared using a t-test for two 
independent samples. Table 21, above, shows that the two groups were similar. The average 
SOL English/Reading Scale Score is well below the state’s passing score of 400. As 
mentioned earlier, baseline scores for some of the measures were missing for students in some 
schools. Again, the number of students missing data was the same for the two groups. Also, 
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the missing data were always for an entire school. Given that the random assignment was 
implemented within each school (and grade), it is very unlikely that there may have been any 
differences in the distributional characteristics of missing data between the treatment and 
control groups. 


Impact of PRJ Participation on Student Reading Achievement 


Impact on all students 


The estimated impact of participation in PRJ on the reading achievement of all students in 
grades 7 and 8 is described in Table 22. Clearly, the null hypothesis of no effect cannot be 
rejected. Students in the treatment and control groups performed equally well on both the 
GMRT and the SOL at the end of the year. After adjusting for the student level covariates that 
were retained in the final analysis model, the difference between the two groups on all four test 
scores was less than two scale score points. Similarly, the effect sizes were virtually zero. 
(The final analysis models, including the intra class correlations that represent the level of 
between-school variation, are presented in Appendix A.) 


Table 22. Impact of PRJ participation on Spring 2011 scores on the GMRT and SOL for 
all students 


Unadjusted Means Adjusted Means Est. 
Test Score 
Control | Treatment | Control | Treatment | Impact 
GMRT Total 502.7 


Reading Extended (289)* 
Scale Score [23.9] ** 


GMRT 
Comprehension 
Extended Scale 
Score 

GMRT Vocabulary 501.0 
Extended Scale (295) 
Score [26.9] 
SOL 398.2 
English/Reading (358) 
Scale Score [50.0] 
* Numbers in parentheses are the numbers of students in each group having valid test scores for both the 2010 and 2011 
spring administrations of each test. 

** Standard deviation 


*** The method used to calculate effect size was Glass’s A, the difference between treatment and control groups’ adjusted 
mean test scores divided by the control group’s test score standard deviation. 


498.2 
(290) 


[31.1] 


Impact on students in each grade 


Although differential effects of the intervention were not expected in the two different grades, 
analyses were carried out to determine whether the results may have been different for students 
in grades 7 and 8. There was more variation in the estimates of impact and effect sizes, but, 
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with one exception, the pattern of non-significant results was repeated. The exception was that 
Grade 7 students in the treatment group did significantly better (p < 0.05) than control students 
on the GMRT Comprehension subtest. The effect size of 0.21 was, however, relatively small. 
Also, after making 12 tests, one would expect almost one of them to be significant at p<0.05 by 
chance. Table 23 displays the results. 


Table 23. Impact of PRJ participation on Spring 2011 scores on the GMRT and SOL 


Unadjusted Means Adjusted Means Est. Effect 
Test Score : 
Control | Treatment | Control | Treatment | Impact | Size*** 


Grade 7 


GMRT Total 497.0 
Reading Extended (152)** 
Scale Score [25.3] 


GMRT 


Comprehension 
Extended Scale (193) 
Score ee 
GMRT Vocabulary 495.7 
Extended Scale (155) 
Score [28.5] 
SOL 402.4 
English/Reading (186) 399.8 


491.8 


Scale Score [49.1] 


Grade 8 


GMRT Total 509.0 
Reading Extended (137)* 508.8 
Scale Score [20.6] ** 


GMRT 
Comprehension 
Extended Scale 
Score 

GMRT Vocabulary 507.0 
Extended Scale (140) 
Score [23.7] 
SOL 393.7 
English/Reading (172) 
Scale Score [50.6] 


505.3 
(137) 
[29.2] 


*Numbers in parentheses are the numbers of students in each group having valid test scores for both the 2010 and 2011 spring 
administrations of each test. 

** Standard deviation 

*** The method used to calculate effect size was Glass’s A, the difference between treatment and control groups’ adjusted 
mean test scores divided by the control group’s test score standard deviation. 
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CONCLUSION 


Research has established a relationship between reading difficulties and poor behavior, 
although the nature of that relationship is unclear (Kos, 1991; McGee, Share, Moffitt, Williams 
& Silva, 1988). For struggling adolescent readers, behavior problems are intensified by the 
consequences of years of reading failure. This failure can be caused by weak phonics skills, 
dysfluency from poor word recognition, broad knowledge deficits, and lack of practice in 
applying comprehension strategies over time. As a result, struggling adolescent readers have 
little confidence in their ability to succeed in reading and little sense of themselves as readers 
(Collins, 1996; Guthrie, Alao, & Rinehart, 1997). Not only are they aware of their reading 
problems but they are likely to have anxiety, low motivation for learning, and lack of self- 
efficacy (Wigfield & Eccles, 1994). For these reasons, striving readers are more comfortable 
“acting out” than publicizing their lack of reading proficiency. 


To address the deficits in the earlier years, Torgensen (2005) suggests that adolescent 
struggling readers need “powerful instruction;” that is, instruction that has the following 
qualities: explicit, systematic, intensive and supportive, scaffolded and attentive to time on 
task, and includes flexible grouping structures, ongoing progress monitoring, and motivational 
techniques (Torgensen, 2005). PRJ J and I/ incorporates each of these features in varying 
degrees, as they use a number of strategies to engage struggling readers, including explicit 
instruction, engaging reading activities, peer-to-peer collaboration and student-centered 
technology. The two-week Expedition is designed with cumulative review to help scaffold 
student learning over time. However, looking at the study findings, PRJ appears to have no 
impact on students’ academic performance, as measured by standardized assessments. 


Reasons for such results are difficult to determine, particularly in a one-year study. 
Unfortunately the abrupt ending of funding for the study cut short the evaluators’ ability to 
pursue the lines of inquiry that appeared as the implementation data started to be analyzed. For 
instance, an unexpected finding in the study was that the interventionists in schools with 90- 
minute lessons covered less material than those in schools with 50-minute lessons. The extent 
to which longer classroom time impacts highly structured interventions is not clear, but the 
question merits exploration, as more schools move into block time. Another question that 
merits further investigation is the relationship between student behavior and highly structured 
interventions. Does a prescriptive and repetitive intervention hinder student behavior? If so, 
what can be done to balance the need for explicit and systematic instruction with the need to 
keep students connected to task? 


A factor to be considered for both the implementation and the impact analysis is that one year 
may not be enough time to learn and apply an intervention. The interventionists learned the 
intervention in August and started work in September. They were experimenting with this new 
knowledge while applying it. The developer and VDOE provided supports to ensure that the 
intervention was implemented as faithfully to the model as possible, but the reality of school 
life intervened in the plans. A number of classes were cancelled, many classes were curtailed 
for other activities, and student behavior may have been an unexpected factor that required 
extra training and support. The site visits from the evaluators showed that the interventionists 
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were being faithful to PRJ’s structure and format, while struggling to better understand all its 
implication, particularly how to balance students’ needs with the need to keep a brisk pacing. 
Looking at Table 14 (p. 39), only one interventionist was able to finish the lessons within the 
allotted time, as recommended by the publisher. However, these lessons appeared to have been 
finished to the detriment of some essential features, such as correction procedures (Table 17, 
p.43). Additionally, as Table 15 (p. 41) indicates, no interventionist was able to finish the 
curriculum by the end of the school year, much less before the assessments used for the impact 
study were conducted. 


It is common sense to suppose that even the strongest program cannot teach itself, as it is 
dependent on a knowledgeable and skillful interventionist who can deliver the program with 
both fidelity and sound judgment. For example, a main classroom management strategy in 
PR/J is to anticipate student behavior problems by keeping a brisk pace of delivery. “Time on 
task” means providing opportunities for students to read, increasing time spent on meaningful 
activities, minimizing time spent on transitions, and managing behavior. These “time on task” 
indicators seem to rely more heavily on teacher judgment then many of the other instructional 
features included in the intervention. In the interviews, the interventionists commented that it 
was difficult to simply forge ahead with a lesson if they felt their students were not grasping 
the content. As one interventionist noted, “J am behind on pacing because of testing and 
student behaviors, but mostly I refuse to move at such a quick pace if the students do not 
understand the materials.” One wonders if teachers were able to make informed decisions as 
needs arose, while maintaining a “brisk” pace throughout it all, particularly as they were 
learning and implementing the intervention at the same time. As the interventionists matured 
in their own learning processes about the intervention and the students, the second 
implementation year would certainly differ. How much this maturation process would 
influence test results is a question that cannot be answered, as the study came to an abrupt end. 
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APPENDIX A: Final Model Results for Impact Analyses 


Table Al. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 GMRT Total Reading Extended Scale Scores for All Students 


Fixed Effects 

Level Effect Estimate | Std Error df t-value Pr > |e| 

School | Intercept 502.06 1.27 8 394.42 0.000 

Student | Journeys 1.52 7 563 0.86 0.390 
Grade 8 6.17 1.81 563 3.40 0.001 
2010 GMRT Total Reading * 0.42 0.04 563 8.93 0.000 
Missing Data Dummy Variable * 197.24 22.09 563 9.45 0.000 

Random Effects 

Level Variance Components Estimate ICC 

School _| Intercept 0.22 0.027 

Student | Level-1 442.99 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 


Table A2. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 


Spring 2011 GMRT Comprehension Extended Scale Scores for All Students 


Fixed Effects 

Level Effect Estimate | Std Error df t-value Pr > |t| 

School | Intercept 498.00 2.12 8 235.19 0.000 

Student | Journeys 1.54 2.33 563 0.66 0.508 
Grade 8 5.01 2.41 563 2.50 0.013 
Gender 3.94 2.34 563 1.68 0.093 
2010 GMRT Total Reading * 0.38 0.06 563 6.47 0.000 
Missing Data Dummy Variable * 182.93 29.76 563 6.15 0.000 

Random Effects 

Level Variance Components Estimate ICC 

School | Intercept 14.86 0.046 

Student | Level-1 768.78 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 
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Table A3. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 GMRT Vocabulary Extended Scale Scores for All Students 


Fixed Effects 

Level Effect Estimate | Std Error df t-value Pr > {t| 

School | Intercept 500.66 1.56 8 321.65 0.000 

Student | Journeys 1.80 2.06 579 0.88 0.382 
Grade 8 6.57 2.13 579 3.09 0.003 
Gender -2.82 2.07 579 -1.36 0.174 
2010 GMRT Total Reading * 0.46 0.05 579 9.00 0.000 
Missing Data Dummy Variable * 223.75 25.66 579 8.72 0.000 

Random Effects 

Level Variance Components Estimate ICC 

School | Intercept 1.97 0.010 

Student | Level-1 619.00 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 


Table A4. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 


Spring 2011 SOL English/Reading Scale Scores for All Students 


Fixed Effects 

Level Effect Estimate | Std Error df t-value Pr > |t| 

School | Intercept 396.96 3.98 8 99.82 0.000 

Student | Journeys 3.13 3.53 696 0.89 0.376 
Grade 8 -11.60 3.65 696 3.18 0.002 
2010 GMRT Total Reading * 0.75 0.09 696 8.47 0.000 
Missing Data Dummy Variable * 358.81 43.75 696 8.20 0.000 

Random Effects 

Level Variance Components Estimate ICC 

School _| Intercept 85.76 0.028 

Student | Level-1 2176.20 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 
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Table A5. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 GMRT Total Reading Extended Scale Scores for Grade 7 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > {t| 
School | Intercept 495.81 1.79 8 276.46 0.000 
Student | Journeys 4.38 2.53 295 1.73 0.084 
2010 GMRT Total Reading * 0.55 0.07 295 8.28 0.000 
Missing Data Dummy Variable * 266.15 32.95 295 8.08 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School _| Intercept 0.002 | 0.015 
Student | Level-1 476.25 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 


Table A6. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 GMRT Comprehension Extended Scale Scores for Grade 7 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > |t| 
School | Intercept 490.59 2.44 8 201.40 0.000 
Student | Journeys 6.67 3.20 297 2.08 0.038 
2010 GMRT Total Reading * 0.56 0.09 297 6.62 0.000 
Missing Data Dummy Variable * 269.35 42.26 297 6.37 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School | Intercept 6.84 | 0.066 
Student | Level-1 764.26 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 
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Table A7. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 


Spring 2011 GMRT Vocabulary Extended Scale Scores for Grade 7 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > {t| 
School | Intercept 494.68 2.28 8 217.04 0.000 
Student | Journeys 3.00 3.02 304 0.99 0.322 
2010 GMRT Total Reading * 0.55 0.08 304 6.89 0.000 
Missing Data Dummy Variable * 274.14 39.74 304 6.90 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School _| Intercept 4.99 0.009 
Student | Level-1 695.95 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 


Table A8. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 SOL English/Reading Scale Scores for Grade 7 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > |t| 
School | Intercept 399.81 6.63 8 60.29 0.000 
Student | Journeys -0.64 4.66 361 -0.14 0.892 
2010 GMRT Total Reading * 0.79 0.12 361 6.34 0.000 
Missing Data Dummy Variable * 391.34 60.92 361 6.42 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School | Intercept 296.32 0.089 
Student | Level-1 1963.52 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 
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Table A9. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 GMRT Total Reading Extended Scale Scores for Grade 8 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > {t| 
School | Intercept 508.75 1.86 8 271.20 0.000 
Student | Journeys -0.94 2.42 265 -0.39 0.696 
2010 GMRT Total Reading * 0.29 0.06 265 5.08 0.000 
Missing Data Dummy Variable * 132.67 29.07 265 4.56 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School _| Intercept 4.08 0.053 
Student | Level-1 389.71 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 


Table A10. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 GMRT Comprehension Extended Scale Scores for Grade 8 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > |t| 
School | Intercept 505.76 2.78 8 182.03 0.000 
Student | Journeys -3.61 3.38 264 -1.07 0.287 
2010 GMRT Total Reading * 0.26 0.08 264 326 0.000 
Missing Data Dummy Variable * 126.78 40.97 264 3.10 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School | Intercept 14.86 0.051 
Student | Level-1 759.93 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 
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Table A11. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 


Spring 2011 GMRT Vocabulary Extended Scale Scores for Grade 8 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > {t| 
School | Intercept 506.61 2.02 8 250.27 0.000 
Student | Journeys 1.16 2.78 273 0.42 0.677 
2010 GMRT Total Reading * 0.38 0.06 273 5.88 0.000 
Missing Data Dummy Variable * 177.76 32.86 273 5.41 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School _| Intercept 0.35 0.28 
Student | Level-1 531.61 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 


Table A12. Final Model Results and Intra Class Correlation for Analysis of Impact of Journeys on 
Spring 2011 SOL English/Reading Scale Scores for Grade 8 Students 


Fixed Effects 
Level Effect Estimate | Std Error df t-value Pr > |t| 
School | Intercept 394.18 3.94 8 99.96 0.000 
Student | Journeys 7A7 5.23 332 1.43 0.154 
2010 GMRT Total Reading * 0.72 0.13 332 5,715 0.000 
Missing Data Dummy Variable * 331.13 62.54 332 5.29 0.000 
Random Effects 
Level Variance Components Estimate ICC 
School | Intercept 18.77 0.014 
Student | Level-1 2282.98 


* Missing data for the baseline covariate were coded with a zero and a zero-one dummy variable was set to one 
for each case missing the baseline covariate. Because of missing data issues in certain schools, the 2010 


GMRT Total Reading score was used as the baseline covariate for all analyses. 
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APPENDIX B: Implementation Study Protocols 


A. Interventionists Check-In Protocol 


School 


(Check the school where the interventionist you are calling works) 


Roanoke 


Norfolk 


Richmond 


Lucy Addison 


Azalea Gardens 


Lucille M. Brown 


James Breckinridge 


Blair 


James Madison 


Norview 


Stonewall Jackson 


Woodrow Wilson 


Date of Call 


(Please write the day in the corresponding month of the call) 


2010 2011 
September January 
October February 
November March 
December April 
May 


Note: Long interview with interventionists in June 


Caller: 


Professional Development 


1. What Voyager-related professional development (PD) have you attended this month? 


None 
Online training: 


Other (specify) 


Adolescent literacy training: 


(number of hours) 


(number of hours) 


2. Ifthe answer to question | is no, why? 


Lack of time 


Lack of interest 


Other reasons (specify) 


No PD was offered this month 
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Coaching Support 


3. Did you meet with the Voyager Implementation Specialist this month? 


Yes (go to question 4) No (go to question 6) 


4. If yes, how many hours of coaching support did you receive this month? 
5. What was the focus of the meeting? 
6. Did you meet with the Project Director this month? 
Yes (go to question 7) No_____— (go to question 10) 
7. If yes, how many hours of coaching support did you receive this month? 
8. What was the focus of the meeting? 


9. How would you rate the coaching or mentoring support you received this month? 


Not relevant Very relevant 
1 2 3 4 5 
Insufficient Sufficient 
1 2 3 4 5 


10. How aligned is the training and support you received from the coach, the project director, the 
liaison, and any other Voyager staff? 
Conflicting Very aligned 
1 2 3 4 5 
11. If you did not meet with the coach, why not? 


a. Iam meeting the coach on 


b. The coach did not come this month 
c. Iwas too busy to spend time with the coach 


d. Ihave been absent from school 


e. Other reasons (Specify) 


12. Did you receive a visit this month from 
a. The School Division Liaison? 


b. Other 


13. What was the focus of the meeting? 


7" grade instruction (the next four questions are related to your 7" grade classes) 


14. How was attendance in your classes this month? 


Almost all students attended 


Number of students chronically absent (more than once a week) 
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____ Students were suspended (OSS) for days 
____ Students were suspended (ISS) for days 
The school was closed for days 

Thad class days cancelled due to 


15. What Expedition and Lesson are you in in each of your 7" grade classes? 


Expedition lesson 


16. Did you use the 11" day for the Expedition this month? 
Yes___ss (go to 12) No_____ (go to 13) 
17. If yes, what activities did you chose for the 11" day? 
a. Re-teaching 
b. Independent work 
c. Other (specify) 


18. How did you make the choice about the 11" day? 


8" grade instruction (the next four questions are related to your 8" grade classes) 


19. How was attendance in your classes this month? 


Almost all students attended 


Number of students chronically absent (more than once a week) 


____ Students were suspended (OSS) for days 
____ Students were suspended (ISS) for days 
The school was closed for days 

I had class days cancelled due to 
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20. What Expedition and Lesson are you in in each of your 8" grade classes? 


Expedition lesson 


21. Did you use the 11" day for the Expedition this month? 
Yes__s (gov to. 17) No____ (go to 18) 
22. If yes, what activities did you chose for the 11" day? 
d. Re-teaching 
e. Independent work 
f. Other (specify) 


23. How did you make the choice about the 11" day? 


Use of time 
24. Can you say that at least 80% of your time this month has been dedicated to teaching Voyager? 
Yes No 


25. If not, what other duties have you been asked to do? 


26. Do you have scheduled time to plan your classes during the school day? 


Yes No 


Conclusion 
27. Is there anything you would like to share with us at this time about the implementation of the 


program? 
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2. Coaches Interview Protocol 


Interview with 
Date 
Site 


1. How frequently do you meet with the teachers? 


2. What is the main focus of her work with the teachers? 


3. What issues the teachers are encountering? 


4. How much support are they receiving from principals? 


5. What factors do you feel influence the implementation of the intervention? 


6. Other comments? 
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3. Classroom Observation Rubric 


Instructions to observer 


1. Observers should stay for the whole class; if you need to leave the room before the end of the 


class, please check here and explain the reason in the back of this page 


2. Before starting the observation, 


a. Ask the teacher the number of the Expedition you will observe. 

b. Within the Expedition, ask the teacher the number of the lesson she is teaching that day 
(from 1 to 10 in the Expeditions sequence). 

c. Ifthe lesson you are observing is lesson 2, 4, and 7 in the Expeditions sequence, then 
you need to ask the teacher the lesson number for the word study she will be teaching 
that day. 

3. If you answer partially or no to any item in your observation forms, please use the back of the 
page to explain your answer. 

4. In the classroom behavior/management table, if the behavior is not applicable during part of the 
observation time, write n/a and deduce that period from the score (e.g. instead of dividing by 5, 
divide by 4 or what is applicable). 

5. Use back of the observation page to enter comments. You don’t need to comment in every 
aspect of your observation but make short comments about behaviors or events that catch your 
attention and can be relevant to a better understanding of why the lesson occurred the way it 


did. 


Note 
VSRII has 11-day Expeditions with two day 10 
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OVERVIEW 


School: Grade 


Observer: 


Time: 


Lesson Number: 


Was the entire lesson completed in the class period? 


In addition to the teacher, is there another adult in the room? 


Who? (circle) Special education teacher 


School administrator 


School district staff 


Observation Date: 


Expedition Number: 


No. of students: 


No 


Teacher Name: 


Observation 


Special education aide 


Voyager coach 


Virginia Project Director 
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Other 
A. Classroom Environment 
es Yes Partially No 
(complete at beginning of lesson) 
1. Teachers have sufficient space to conduct individual and/or 
group work 
2. Instructional areas are clearly identified (i.e. whole group, 
independent small group, word study) 
3. Teacher resources for the daily lesson are readily available 
4. All students have readily available materials, as needed 
B. Lesson Planning and Delivery — overview ; Not 
Yes Partially No 
(complete at the end of lesson) observed 
5. Teacher closely follows the curriculum guide during 
instruction 
6. Pace is brisk and business-like, yet personal 
7. Skills are modeled correctly 
The steps of the correction procedures are followed as 
needed 
9. Teacher puts students into groups as indicated by the 
lesson 
10. Teacher uses built-in differentiated instruction 
strategies as needed : 
[| re-teach lesson 
L| word study lesson 
| English Language Learner strategies 
[| challenge questions 
[| Paired reading 
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Lesson 1/Lesson 6 


C. Lesson Planning and Delivery- Lesson 
Specific 
(Check the box next to each activity you observe ) 


Components 
delivered in 
order? 
(Y/N) 


Steps 
Delivered 
in order? 

(Y/N) 


Components 
delivered 
within 
allotted time? 
(Y/N) 


WHOLE GROUP 


Component 1: Advanced Word Study (10 min.) 
Affixes 

Sight Words 

Multisyllabic Words 


Spelling 


Fluency 


Component 2: Before Reading (15 min.) 


Expedition Introduction (lesson 1 only) 


Review content with DVD (lesson 6 only) 


Vocabulary 


Comprehension Strategies 


Component 3: Reading (20 min.) 


Read Lesson Passage 


Model Vocabulary and Comprehension 
Strategies 


Component 4: After Reading (5 min.) 


Review Vocabulary and Comprehension 
Strategies 


D. Classroom Behavior/Management 


Minutes 


10 10 10 


10 


10 


Total 


Half or more of the students are paying attention 
to teacher or following teacher instructions 


Half or more of the students are responding to 
teacher questions or prompts 


Half or more of the students are actively 
participating in the activities assigned by the 
teacher (group or individually) 


Teacher addresses student behavior promptly to 
minimize disruption in the classroom 


a, 


Teacher makes an effort to involve students who 
appear disengaged 


6. 


Students follow expectations for working in 
groups 


D. Classroom Behavior/Management Scale: 0 Not At All 1 Occasionally 2 Frequently 
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Lesson 2/Lesson 4/Lesson 7 


C. Lesson Planning and Delivery- Lesson Specific 
(Check the box next to each activity you observe in the 
lesson) 


Components 
delivered in 
order? 
(Y/N) 


Steps 
delivered 
in order? 

(Y/N) 


Components 
delivered within 
allotted time? 
(Y/N) 


WHOLE GROUP 
Component 1: Before Reading (10 min.) 


Vocabulary Review 


Applied Vocabulary 


Component 2: Reading (15 min.) 


Reread Lesson 1, 3, or 6 Passage 


Practice Comprehension Strategies 


INDEPENDENT SMALL GROUP* 
Component 3: After Reading (10 min.) 


Writing in Response to Reading 


Component 4: Paired Fluency Practice (15 min.) 
Component 5: Passports Reading Journeys Library 


Building Fluency 


Self-selected Texts 


*During this 25-minute period, a small group of students may be involved in teacher-directed Word 
Study while the other students work independently. Use the Word study form to observe the small group. 
If no small group happened, check here and use the back of the form to explain why. 


D. Classroom Behavior/Management 


Minutes 


Total 


10 10 


10 10 


10 


1. Half or more of the students are paying attention to 
teacher or following teacher instructions 


2. Half or more of the students are responding to 
teacher questions or prompts 


3. Half or more of the students are actively 
participating in the activities assigned by the teacher 
(group or individually) 


4. Teacher addresses student behavior promptly to 
minimize disruption in the classroom 


5. Teacher makes an effort to involve students who 
appear disengaged 


6. Students follow expectations for working in groups 


D. Classroom Behavior/Management Scale: 


0 Not At All 
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1 Occasionally 2 Frequently 
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Lesson 3/ Lesson 8 


C. Lesson Planning and Delivery- Lesson Specific 
(Check the box next to each activity you observe in the 
lesson) 


Components 
delivered in 
order? 
(Y/N) 


Steps 
delivered 
in order? 

(Y/N) 


Components 
delivered 
within 
allotted time? 


(Y/N) 


WHOLE GROUP 


Component 1: Advanced Word Study (10 min.) 
Affixes 

Sight Words 

Multisyllabic Words 


Spelling 


Fluency 


Component 2: Before Reading (15 min.) 


Vocabulary 


Comprehension Strategies 


Component 3: Reading (15 min.) 


Read Lesson 3 or 8 Passage 


Model Vocabulary and Comprehension 
Strategies 


Component 4: After Reading (5 min.) 


Review Vocabulary and 
Comprehension Strategies 


D. Classroom Behavior/Management 


Minutes 


10 10 


10 10 


10 


Total 


Half or more of the students are paying attention to 
teacher or following teacher instructions 


Half or more of the students are responding to 
teacher questions or prompts 


Half or more of the students are actively 
participating in the activities assigned by the 
teacher (group or individually) 


Teacher addresses student behavior promptly to 
minimize disruption in the classroom 


Teacher makes an effort to involve students who 
appear disengaged 


6. 


Students follow expectations for working in groups 


D. Classroom Behavior/Management Scale: 


Virginia Striving Readers Intervention Initiative, Year One Implementation 
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Lesson 9 


C. Lesson Planning and Delivery- Lesson Specific 
(Check the box next to each activity you observe in the 
lesson) 


Components 
delivered in 
order? 
(Y/N) 


delivered 


Steps 


in order? 


(Y/N) 


Components 


delivered within 
allotted time? 
(Y/N) 


WHOLE GROUP 


Component 1: Before Reading (10 min.) 


Vocabulary Review 


Applied Vocabulary 


Component 2: Reading (10 min.) 


Reread Lesson 8 Passage 


Component 3: After Reading (10 min.) 


Expedition Review 


Component 4: Assessment (15 min.) 


Tips for Success 


Comprehension and Vocabulary Assessment 
(New Reading Passage/High-Stakes Practice) 


Component 5: Expedition Wrap-Up (5 min.) 


DVD Closing Segment 


Discuss Probing Questions 


D. Classroom Behavior/Management 


Minutes 


10 10 


10 


10 


10 


Total 


Half or more of the students are paying attention to 
teacher or following teacher instructions 


Half or more of the students are responding to 
teacher questions or prompts 


Half or more of the students are actively 
participating in the activities assigned by the teacher 
(group or individually) 


Teacher addresses student behavior promptly to 
minimize disruption in the classroom 


Teacher makes an effort to involve students who 
appear disengaged 


6. 


Students follow expectations for working in groups 


D. Classroom Behavior/Management Scale: 0 Not At All 
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1 Occasionally 2 Frequently 
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Word Study Sequence Lesson 1-4, 6-9 Form 


Components Steps Components 

delivered in | delivered | delivered within 
order? in order? allotted time? 
(Y/N) (Y/N) (Y/N) 


E. Lesson Planning and Delivery — Word Study 
(Check the box next to each activity you observe in the 
lesson) 


SMALL GROUP (25 min.) 
[| Fluency 


[1 Phonics 


Sight Words 


Spelling 


Passage Reading and Comprehension 


Vocabulary 
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